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(54) Method and apparatus for detecting structural elements of subjects in digital images 



(57) Structural element candidates, estimated to be 
predetermined structural elements of a predetermined 
subject, are detected from an image that includes sub- 
jects. The subjects that include the structural element 
candidates are detected from the image in the vicinity of 
the detected structural element candidates. The charac- 
teristics of the structural elements are discriminated from 



the image in the vicinity of the structural element candi- 
dates, at a higher accuracy than when the structural el- 
ements were detected, from among the characteristics 
of the image in the vicinities of structural element candi- 
dates included within the detected subjects, so as to con- 
firm the structural element candidates as being the pre- 
determined structural elements. 
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Description 

BACKGROUND OF THE INVENTION 
Field of the Invention 

[0001] The present invention relates to a structural el- 
ement detecting method, a structural element detecting 
apparatus, and a structural element detecting program, 
for detecting predetermined structural elements, which 
are included in predetermined subjects, from within im- 
ages that include the predetermined subjects. 

Description of the Related Art 

[0002] There are cases in which pupils (or portions of 
pupils) of people or animals, photographed by flash pho- 
tography at night or in dark places, are photographed as 
being red or gold. For this reason, various methods for 
correcting the color of pupils, which have been photo- 
graphed as being red or gold (hereinafter, cases in which 
pupils are photographed as being gold are also referred 
to as "red eye"), to normal pupil colors by digital image 
processing have been proposed. 

[0003] For example, Japanese Unexamined Patent 
Publication No. 2000-013680 discloses a method and 
apparatus for automatically discriminating red eyes. This 
method and apparatus automatically discriminate red 
eyes based on colors, positions, and sizes of pupils within 
a region specified by an operator. Japanese Unexamined 
Patent Publication No. 2001 -148780 discloses a method 
wherein: predetermined characteristic amounts are cal- 
culated for each pixel within a region specified by an op- 
erator; and portions having characteristics that corre- 
spond to pupil portions are selected as targets of correc- 
tion. However, in discriminating processes which are 
based solely on characteristics of pupil portions, it is dif- 
ficult to discriminate targets having local redness, such 
as red lighting, from red eyes. For this reason, it is difficult 
for this process to be executed automatically, without op- 
erator intervention. 

[0004] On the other hand, Japanese Unexamined Pat- 
ent Publication No. 2000-125320 discloses a method 
wherein: faces are detected first; and red eye detection 
is performed within regions detected to be faces. In this 
method, false positives, such as red lights being detected 
as red eyes, does not occur. However, if errors occur 
during face detection, red eyes cannot be accurately de- 
tected. Therefore, the accuracy of the facial detection 
becomes an issue. 

[0005] The simplest method for detecting faces is to 
detect oval skin colored regions as faces. However, peo- 
ple's faces are not necessarily uniform in color. There- 
fore, it is necessary to broadly define "skin color", which 
is judged to be the color of faces. However, the possibility 
of false positive detection increases in the case that the 
range of colors is broadened in a method that judges 
faces based only on color and shape. For this reason, it 



is preferable that faces are judged utilizing finer charac- 
teristics than just the color and the shapes thereof, in 
order to improve the accuracy of facial detection. How- 
ever, if characteristics of faces are extracted in detail, the 
5 time required for facial detection processes greatly in- 
creases. 

[0006] That is, the method disclosed in Japanese Un- 
examined Patent Publication No. 2000-125320 is capa- 
ble of detecting red eyes with high accuracy, yet gives 
10 no consideration to processing time. In the case that the 
method is applied to an apparatus having comparatively 
low processing capabilities (such as a low cost digital 
camera), the apparatus cannot function practically. 

15 SUMMARY OF THE INVENTION 

[0007] The object of the present invention is to provide 
a method, apparatus, and program which are capable of 
accurately detecting structural elements (such as eyes 

20 and red eyes) of predetermined subjects (such as faces 
of living organisms, for example, humans and animals) 
from within images that include the predetermined sub- 
jects, with a small amount of calculations in a short period 
of time. To achieve this object, the present invention pro- 

25 vides processes, to be described below, to be performed 
by a structural element detecting apparatus, a structural 
element detecting method, and a structural element de- 
tecting program. Note that in the description below, "ap- 
paratus" includes semiconductor devices, such as image 

30 processors, in addition to conventional devices. In addi- 
tion, the term "computer" refers not only to personal com- 
puters and servers, but also includes any device capable 
of executing programs, that is, having calculating means 
and memory means. 

35 [0008] The first structural element detecting apparatus 
of the present invention is a structural element detecting 
apparatus for detecting predetermined structural ele- 
ments included in predetermined subjects from within im- 
ages that include the subjects, characterized by compris- 

40 ing: a structural element candidate detecting section; a 
subject detecting section; and a structural element con- 
firming section. The structural element candidate detect- 
ing section detects structural element candidates, by dis- 
criminating characteristics inherent to the predetermined 

45 structural elements, from among characteristics of the 
images. The subject detecting section detects subjects 
that include the structural element candidates, by dis- 
criminating characteristics inherent to the predetermined 
subjects, from among characteristics of the images in the 

50 vicinities of the structural elements candidates. The 
structural element confirming section confirms that the 
structural element candidates are the predetermined 
structural elements, by discriminating characteristics in- 
herent to the predetermined structural elements with a 

55 higher degree of accuracy than that of the structural el- 
ement candidate detecting section, from among the char- 
acteristics of the images in the vicinities of the structural 
element candidates, which are included within subjects 
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detected by the subject detecting section. 
[0009] The first structural element detecting method of 
the present invention is a structural element detecting 
method for detecting predetermined structural elements 
included in predetermined subjects from within images 
that include the subjects, characterized by comprising: a 
first structural element candidate detecting process; a 
first subject detecting process; and a structural element 
confirming process. The first structural element candi- 
date detecting process detects structural element candi- 
dates, by discriminating characteristics inherent to the 
predetermined structural elements, from among charac- 
teristics of the images. The first subject detecting process 
detects subjects that include the structural element can- 
didates, by discriminating characteristics inherent to the 
predetermined subjects, from among characteristics of 
the images in the vicinities of the structural element can- 
didates. The structural element confirming process con- 
firms that the structural element candidates are the pre- 
determined structural elements, by discriminating char- 
acteristics inherent to the predetermined structural ele- 
ments with a higher degree of accuracy than that of the 
structural element candidate detecting section, from 
among the characteristics of the images in the vicinities 
of the structural element candidates, which are included 
within the subjects detected by the subject detecting sec- 
tion. 

[0010] The structural element detecting program of the 
present invention is a program that provides a function 
of detecting predetermined structural elements included 
in predetermined subjects from within images that in- 
clude the subjects. The program is characterized by 
causing a computer to execute: a first structural element 
candidate detecting process; a first subject detecting 
process; and a structural element confirming process. 
The first structural element candidate detecting process 
detects structural element candidates, by discriminating 
characteristics inherent to the predetermined structural 
elements, from among characteristics of the images. The 
first subject detecting process detects subjects that in- 
clude the structural element candidates, by discriminat- 
ing characteristics inherent to the predetermined sub- 
jects, from among characteristics of the images in the 
vicinities of the structural element candidates. The struc- 
tural element confirming process confirms that the struc- 
tural element candidates are the predetermined structur- 
al elements, by discriminating characteristics inherent to 
the predetermined structural elements with a higher de- 
gree of accuracy than that of the structural element can- 
didate detecting section, from among the characteristics 
of the images in the vicinities of the structural element 
candidates, which are included within the subjects de- 
tected by the subject detecting section. 
[0011] In the structural element confirming process, 
data regarding faces detected by the face detecting sec- 
tion may be utilized, in order to perform discrimination at 
a higher degree of accuracy that that of the structural 
element candidate detecting process. Specifically, char- 



acteristics inherent to structural elements having the 
same orientations as those of detected subjects may be 
discriminated from among the characteristics of images 
in the vicinities of structural element candidates, and 

5 structural element candidates having such characteris- 
tics may be confirmed to be the predetermined structural 
elements, for example. Alternatively, characteristics in- 
herentto structural elements having sizes estimated from 
the sizes of detected subjects may be discriminated, and 

10 structural element candidates having such characteris- 
tics may be confirmed to be the predetermined structural 
elements. As a further example, characteristics inherent 
to structural candidates having the same orientations as 
those of detected subjects and which are of sizes esti- 

15 mated from the sizes of detected subjects may be dis- 
criminated, and structural element candidates that satisfy 
predetermined conditions regarding the orientations and 
sizes thereof may be confirmed to be the predetermined 
structural elements. If the orientations and sizes are lim- 

20 jted, the target ranges for discrimination are narrowed. If 
structural element candidates are discriminated within 
the narrow target ranges, the probability that these struc- 
tural element candidates are the predetermined structur- 
al elements is high. 

25 [001 2] The reliability of detection results can be further 
improved, by performing the following processes after 
the structural element candidates are confirmed to be 
the predetermined structural elements. First, the number 
of structural element candidates, detected by a first struc- 

30 tural element candidate detecting process and confirmed 
as being the predetermined structural elements, are com- 
pared against the maximum number of structural ele- 
ments which are possibly included within the subjects 
detected by the subject detecting section. In the case 

35 that the subjects are front facing faces, and the structural 
elements to be detected are red eyes, a maximum of two 
red eyes are possibly included in the front facing faces. 
In this case, the maximum number of structural elements, 
which are possibly included in the subject, is 2. 

40 [0013] A second structural element candidate detect- 
ing process, having a higher detection rate than the first 
structural element candidate detecting process, is exe- 
cuted in the case that number of structural element can- 
didates, which have been confirmed to be structural el- 

45 ements by the structural element confirming section, is 
less than the maximum number. Then, the structural el- 
ement confirming process is executed with respect to the 
newly detected structural element candidates. 
[001 4] It is desirable that the confirmation of the struc- 

50 tural elements is not limited to judging whether the struc- 
tural element candidates are the predetermined structur- 
al elements. That is, it is desirable that accurate positions 
of the structural elements are specified and confirmed, 
based on information obtained during the discrimination 

55 processes. 

[001 5] It is desirable that the structural element candi- 
date detecting section is capable of executing a plurality 
of structural element candidate detecting processes, 
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which have different detection rates. In this case, it is 
desirable that a second structural element candidate de- 
tecting process, having a higher detection rate than a 
first structural element candidate detecting process, is 
executed in the case that number of structural element 
candidates, which have been detected by the first struc- 
tural element candidate detecting process and confirmed 
to be the predetermined structural elements by the struc- 
tural element confirming section, is one or less. Further, 
it is desirable that the subject detecting process and the 
structural element confirming process are executed with 
respect to the structural element candidates, which have 
been detected by the second structural element candi- 
date detecting process. 

[0016] The structural element detecting process real- 
ized by the method, apparatus, and program for structural 
element detection of the present invention comprises the 
following steps. First, structural element candidates are 
detected in the structural element candidate detecting 
step. Then, the vicinities of the structural element candi- 
dates are set to be search ranges for detecting structural 
elements in the structural element confirming step. The 
structural element confirming step discriminates the 
structural elements with higher accuracy than in the 
structural element candidate detecting step, by utilizing 
information regarding the detected subjects, for example. 
Therefore, the accuracy in judging whether the structural 
element candidates are the predetermined structural el- 
ements is higher than the detection accuracy during the 
structural element candidate detecting step. Thereby, 
structural element candidates detected during the struc- 
tural element candidate detecting step, which are false 
positive detections, can be eliminated. 
[0017] Highly accurate judgment is performed during 
the structural element confirming step, by obtaining more 
information during the discrimination process than during 
the structural element candidate detecting step. There- 
fore, the positions of the structural element candidates 
may be accurately specified by utilizing the obtained in- 
formation. 

[001 8] The apparatus, method and program described 
above execute processes to detect the structural ele- 
ments and the subjects, based on characteristic amounts 
included in images. Specifically, the first structural ele- 
ment candidate detecting process, the first subject de- 
tecting process, and the highly accurate structural ele- 
ment detecting process to confirm the structural ele- 
ments, are performed. In afurther preferred embodiment, 
the second structural element candidate detecting proc- 
ess and the second subject detecting process are per- 
formed. 

[001 9] The present invention further provides a second 
apparatus, a second method, and a second program for 
efficiently detecting detection targets, such as structural 
elements and subjects. The second apparatus and the 
second method to be described below may be incorpo- 
rated into the first structural element detecting apparatus 
and the first structural element detecting method of the 



present invention described above. Further, the second 
apparatus and the second method may be applied to 
other apparatuses and methods for detecting detection 
targets. 

5 [0020] The second apparatus of the present invention 
comprises: target region setting means, for setting judg- 
ment target regions within an image; and judging means, 
for judging whether images within the judgment target 
regions set by the target region setting means represent 
10 detection targets (the predetermined structural elements 
or the predetermined subjects), based on a plurality of 
characteristic amounts, which are calculated from pixel 
values of pixels that constitute the images within the judg- 
ment target regions. 
15 [0021] The judging means classifies and records for- 
mulas for calculating the plurality of characteristic 
amounts into N groups (N>1), such that the amount of 
calculations required to obtain characteristic amounts by 
formulas in a K th group (1<K<N) is less than the amount 
of calculations required to obtain characteristic amounts 
by formulas in a K+1 st group. By employing these formu- 
las, the judging means performs judgments as follows. 
[0022] In the case that it is judged that the image within 
the judgment target region does not represent a detection 
target, based on characteristic amounts obtained by for- 
mulas in aK th group, the judging means outputs judgment 
results indicating that an image within a judgment target 
region does not represent the detection target. On the 
other hand, in the case that it is judged that an image 
within a judgment target region represents the detection 
target, based on characteristic amounts obtained by for- 
mulas in a K th group, the judging means performs judg- 
ment, based on characteristic amounts obtained by for- 
mulas in a K+1 st group, with respect to the judgment tar- 
get region. 

[0023] Then, in the case that it is judged that the image 
within the judgment target region represents the detec- 
tion target, based on characteristic amounts obtained by 
formulas in the N th group, the judging means outputs 
judgment results indicating that the image within the judg- 
ment target region represents the detection target. 
[0024] The judging means may perform first judgment, 
based on the characteristic amounts obtained by formu- 
las in a K th group, with respect to a plurality of judgment 
target regions set by the target region setting means; and 
then perform second judgment, based on the character- 
istic amounts obtained by formulas in a K+1 st group, with 
respect to judgment target regions, which have been 
judged to represent the detection targets by the first judg- 
ment. 

[0025] The judgment target regions may be set such 
that the positions of judgment target regions closest to 
each other are shifted at least two pixels, during at least 
a portion of the judgments performed by the judging 
means. 

[0026] The judging means may perform first judgment, 
based on characteristic amounts obtained by formulas 
in a K th group, with respect to judgment target regions, 
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which are set within an image represented at a predeter- 
mined resolution; and then perform second judgment, 
based on characteristic amounts obtained by formulas 
in a K+1 st group, with respect to the judgment target re- 
gions, which are set within the image represented at a 
higher resolution than that of the first judgment. 
[0027] The judging means may perform first judgment, 
based on the characteristic amounts obtained by formu- 
las in a K th group, with respect to the plurality of judgment 
target regions set by the target region setting means; and 
then perform second judgment, based on the character- 
istic amounts obtained by formulas in a K+1 st group, with 
respect to a plurality of judgment target regions, which 
are set such that at least a portion thereof overlaps with 
judgment target regions that have been judged to repre- 
sent red eye by the first judgment. 
[0028] At this time, at least one of the formulas for cal- 
culating characteristic amounts classified in a K th group 
may also be included in a K+1 st group. 
[0029] The construction described above may be de- 
scribed as procedural steps in the following manner. 
[0030] First, formulas for calculating the plurality of 
characteristic amounts are classified and organized into 
N groups (N>1), such that the amount of calculations 
required to obtain characteristic amounts by formulas in 
a K th group (1<K<N) is less than the amount of calcula- 
tions required to obtain characteristic amounts by formu- 
las in a K+1 st group. 

[0031 ] I n the case that it is judged that the image within 
thejudgmenttarget region does not represent a detection 
target, based on characteristic amounts obtained by for- 
mulas in a K th group, judgment results indicating that an 
image within a judgment target region does not represent 
the detection target are output. On the other hand, in the 
case that it is judged that an image within a judgment 
target region represents the detection target, based on 
characteristic amounts obtained by formulas in a K th 
group, judgment is performed, based on characteristic 
amounts obtained by formulas in a K+1 st group, with re- 
spect to the judgment target region. 
[0032] Then, in the case that it is judged that the image 
within the judgment target region represents the detec- 
tion target, based on characteristic amounts obtained by 
formulas in the N th group, judgment results indicating that 
the image within the judgment target region represents 
the detection target are output. 

[0033] The procedural steps described above may be 
described as a program executed by a computer in the 
following manner. 

[0034] Formulas for calculating the plurality of charac- 
teristic amounts are classified and organized into N 
groups (N>1), such that the amount of calculations re- 
quired to obtain characteristic amounts by formulas in a 
K th group (1 <K<N) is less than the amount of calculations 
required to obtain characteristic amounts by formulas in 
a K+1 st group. The program performs judgments by uti- 
lizing these formulas. 

[0035] The program causes a computer to output judg- 



ment results indicating that an image within a judgment 
target region does not represent a detection target, in the 
case that the computer judges that the image within the 
judgment target region does not represent the detection 
5 target, based on characteristic amounts obtained by for- 
mulas in a K th group. 

[0036] On the other hand, in the case thatthe computer 
judges that an image within a judgment target region rep- 
resents the detection target, based on characteristic 

10 amounts obtained by formulas in a K th group, the com- 
puter is caused to perform judgment, based on charac- 
teristic amounts obtained by formulas in a K+1 st group, 
with respect to the judgment target region. 
[0037] Then, in the case that the computer judges that 

15 the image within the judgment target region represents 
the detection target, based on characteristic amounts ob- 
tained by formulas in the N th group, the computer is 
caused to output judgment results indicating that the im- 
age within the judgment target region represents the de- 

20 tection target. 

[0038] In the judging method described above, judging 
processes that are executed often are performed em- 
ploying characteristic amounts that require small 
amounts of calculations. Characteristic amounts, which 

25 are higher in accuracy but require greater amounts of 
calculations, are only utilized in judgment processes tar- 
geting limited regions. Therefore, candidates can be ef- 
ficiently narrowed, and detection targets can be efficiently 
detected, without lowering the accuracy of detection. 

30 

BRIEF DESCRIPTION OF THE DRAWINGS 
[0039] 

35 Figure 1 illustrates the procedures of red eye detec- 
tion in a first embodiment. 

Figure 2 illustrates an example of an image, which 
is a target for red eye detection. 
Figure 3 is an enlarged view of a portion of an image, 
40 which is a target for red eye detection. 

Figure 4 illustrates an example of the definition 
(score table) of the relationship between character- 
istic amounts and scores. 

Figures 5A, 5B, 5C, 5D, and 5E illustrate examples 
45 of red eye learning samples. 

Figure 6 is a flow chart that illustrates N types of 
judging processes. 

Figures 7A and 7B are diagrams for explaining the 
relationship between red eye detection and image 
50 resolution. 

Figure 8 is a diagram for explaining a process which 
is performed with respect to red eye candidates 
which have been redundantly detected. 
Figures 9A and 9B illustrate examples of methods 
55 for calculating characteristic amounts. 

Figure 10 is a flow chart for explaining a second 
method for improving processing efficiency during 
red eye candidate detecting processes. 
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Figure 1 1 is a diagram for explaining a third method 
for improving processing efficiency during red eye 
candidate detecting processes. 
Figures 12A and 12B are diagrams for explaining a 
fourth method for improving processing efficiency 
during red eye candidate detecting processes. 
Figure 13 is a diagram for explaining a fifth method 
for improving processing efficiency during red eye 
candidate detecting processes. 
Figure 14 is a flow chart for explaining a sixth method 
for improving processing efficiency during red eye 
candidate detecting processes. 
Figure 15 is a diagram for explaining scanning of a 
judgment target region during face detecting proc- 
esses. 

Figure 16 is a diagram for explaining rotation of a 
judgment target region during face detecting proc- 
esses. 

Figure 17 is a flow chart that illustrates a face de- 
tecting process. 

Figure 18 is a diagram for explaining calculation of 
characteristic amounts during face detecting proc- 
esses. 

Figure 19 is a diagram for explaining the manner in 
which search regions are set during red eye confirm- 
ing processes. 

Figure 20 illustrates an example of a judgment target 
region, which is set within the search region of Figure 
19. 

Figures 21 A, 21 B, and 21 C illustrate examples of 
search regions, which are set on images of differing 
resolutions. 

Figure 22 is a diagram for explaining a process fro 
confirming the positions of red eyes. 
Figure 23 is a flow chart that illustrates a red eye 
confirming process. 

Figure 24 illustrates an example of a red eye cor- 
recting process. 

Figure 25 is a graph for explaining a pre-process that 
facilitates detection of red regions. 

DESCRIPTION OFTHE PREFERRED EMBODIMENTS 

[0040] Hereinafter, a procedure will be described, in 
which red eyes are detected as structural components 
from images that include human faces as subjects. 

[Outline of Red Eye Detecting Procedure] 

[0041 ] First, the outline of a red eye detecting process 
will be described with reference to Figure 1 and Figure 
2. Figure 1 illustrates the steps of red eye detection. As 
illustrated in Figure 1, the present embodiment detects 
red eyes included in an image 1, by executing a three 
step process, comprising a red eye candidate detecting 
step 2, a face detecting step 3, and a red eye confirming 
step 4. Information representing accurate positions of red 
eyes is obtained, as detection results 5. 



[0042] In the case that the number of red eyes con- 
firmed by the red eye confirming step 4 is one or less 
(none, or only one) even if the face detected by the face 
detecting step 3 is a front facing face, a second red eye 
5 candidate detecting step, which has a higher degree of 
accuracy than the red eye candidate detecting step 2 is 
performed. Then, the face detecting step 3 and the red 
eye confirming step 4 are performed with respect to the 
red eye candidates which have been detected by the 
10 second red eye candidate detecting step. In the present 
specification, the series of processes described above, 
performed in the case that the number of red eyes con- 
firmed by the red eye confirming step 4 is one or less, 
will be referred to as a redetection process. 
15 [0043] Figure 2 illustrates an example of the image 1 . 
The image 1 is a photographic image, in which a person 
has been photographed with red eyes 7a and 7b. A red 
light 7c is also pictured in the photographic image. Here- 
inafter, the outline of the red eye candidate detecting step 
20 2, the face detecting step 3 and the red eye confirming 
step 4 will be described for the case that the image of 
Figure 2 is processed, as an example. 
[0044] The red eye detecting step 2 searches for por- 
tions of the image 1 which are estimated to be red eyes 
25 (red eye candidates) . In cases in which red eye candi- 
dates are found, the positional coordinates of the red eye 
candidates are recorded in a memory medium. Because 
red eyes, of which the sizes and orientations are un- 
known, are to be detected from the entirety of the image 
30 1 in the red eye candidate detecting step 2, processing 
efficiency is prioritized above detection accuracy. In the 
present embodiment, the red eye candidate detecting 
step 2 judges that pupils exist, based only on the char- 
acteristics thereof. For this reason, in the case that the 
35 image of Figure 2 is processed, there is a possibility that 
the light 7c in the background is detected as a red eye 
candidate, in addition to the red eyes 7a and 7b. 
[0045] The face detecting step 3 searches for portions, 
which are estimated to be faces, from within the image 
40 1. However, the search for the faces is performed only 
in the peripheral regions of the red eye candidates, which 
have been detected in the red eye candidate detecting 
step 2. In the case that the red eye candidates are true 
red eyes, faces necessarily exist in their peripheries. In 
45 the case that portions which are likely to be faces are 
found during the face detecting step 3, information, such 
as the size of the face and the orientation thereof, are 
recorded in the memory medium, correlated with the red 
eye candidates that served as the reference points for 
50 the face search. On the other hand, in the case that no 
portions which are likely to be faces are found, informa- 
tion relating to the red eye candidates that served as the 
reference points for the face search is deleted from the 
memory medium. 
55 [0046] In the case that the image of Figure 2 is proc- 
essed, no portion which is likely to be a face is detected 
in the periphery of the light 7c. Therefore, information 
regarding the light 7c is deleted form the memory medi- 
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um. A face 6 is detected in the periphery of the red eyes 
7a and 7b. Accordingly, information relating to the red 
eyes 7a and 7b are correlated with information regarding 
the face 6, and rerecorded in the memory medium. 
[0047] The red eye confirming step 4 judges whether 
the red eye candidates, which have been correlated with 
faces in the face detecting step 3, are true red eyes. In 
the case that the candidates are true red eyes, their po- 
sitions are also accurately confirmed. 
[0048] The red eye confirming step 4 utilizes the results 
of the face detecting step 3. Specifically, information re- 
garding detected faces are utilized to estimate sizes and 
orientations of red eyes, thereby narrowing down regions 
which are likely to be red eyes. Further, the positions of 
red eyes are estimated based on information regarding 
the detected faces. Then, a detection process having 
higher accuracy than that of the red eye candidate de- 
tecting step 2 is executed within limited regions in the 
peripheries of the positions. 

[0049] In the case that red eye candidates are judged 
not to be true red eyes during the red eye confirming step 
4, information relating to the red eye candidates is deleted 
from the memory medium. On the other hand, in the case 
that red eye candidates are judged to be true red eyes, 
the accurate positions thereof are obtained. 
[0050] The positions of red eye candidates are evalu- 
ated utilizing the information regarding the detected faces 
in the red eye confirming step 4. In the case that the red 
eye candidates are located at positions which are inap- 
propriate for eyes within faces, information relating to the 
red eye candidates is deleted from the memory medium. 
[0051 ] For example, in the case that a red circular mark 
is painted on a person's forehead, the red eye candidate 
detecting step 2 will detect the mark as a red eye candi- 
date, and the face detecting step 3 will detect a face in 
the periphery of the mark. However, it will be judged that 
the red eye candidate is located in the forehead, which 
is an inappropriate position for eyes, during the red eye 
confirming step 4. Therefore, information relating to the 
red eye candidate is deleted from the memory medium. 
[0052] In the case of the image of Figure 2, the accu- 
rate positions of the red eyes 7a and 7b are confirmed 
in the red eye confirming step 4. The red eye confirming 
step outputs the positional coordinates of the confirmed 
red eyes as detection results 5. 

[0053] An apparatus for detecting red eyes by the 
above process may be realized by loading a program 
that causes execution of each of the aforementioned 
steps into an apparatus comprising: a recording medium, 
such as a memory unit; a calculating means for executing 
processes defined by the program; and an input/output 
interface for controlling data input from external sources 
and data output to external destinations. 
[0054] Alternatively, an apparatus for detecting red 
eyes by the above process may be realized by incorpo- 
rating a memory/logic device, designed to execute the 
red eye candidate detecting step 2, the face detecting 
step 3, and the red eye confirming step 4, into a prede- 



termined apparatus. 

[0055] In other words, not only general use computers, 
but any apparatus, in which programs or semiconductor 
devices can be loaded, even if they are built for other 
5 specific uses, may function as an apparatus for detecting 
red eyes by the above process. Examples of such appa- 
ratuses are digital photographic printers and digital cam- 
eras. 

10 [Red Eye Candidate Detecting Step] 

[0056] Next, the red eye candidate detecting step 2 (a 
first structural element candidate detecting step) will be 
described in detail. During the red eye candidate detect- 

15 ing step 2, the red eye detecting apparatus first converts 
the color space of an obtained image. Specifically, dis- 
play color systemof the image is converted, by replacing 
the R (red), G (green), and B (blue) values of each pixel 
in the image with Y (luminance), Cb (color difference be- 

20 tween green and blue), Cr (color difference between 
green and red), and Cr* (color difference between skin 
color and red) by use of predetermined conversion for- 
mulas. 

[0057] YCbCr is a coordinate system which is com- 

25 monly utilized in JPEG images. Cr* is a coordinate axis 
that represents a direction in which red and skin color 
are best separated within an RGB space. The direction 
of this coordinate axis is determined in advance, by ap- 
plying a linear discriminant analysis method to red sam- 

30 pies and skin colored samples. By defining this type of 
coordinate axis, the accuracy of judgment, to be per- 
formed later, is improved compared to cases in which 
judgment is performed within a normal YCbCr space. 
[0058] Figure 3 is a magnified view of a portion of the 

35 image 1 , which has been color space converted. The red 
eye detecting apparatus sets a judgment target region 8 
on the image 1, as illustrated in Figure 3. The red eye 
detecting apparatus examines the image within the judg- 
ment target region 8 to determine how many character- 

40 istics of red eye are present therein. In the present em- 
bodiment, the size of the judgment target region 8 is 10 
pixels x 10 pixels. 

[0059] The determination regarding how many char- 
acteristics of red eye are present within the judgment 

45 target region 8 is performed in the following manner. First, 
characteristic amounts that represent likelihood of being 
red eyes, scores corresponding to the value of the char- 
acteristic amounts, and a threshold value are defined in 
advance. For example, if pixel values are those that rep- 

50 resent red, that would be grounds to judge that red eye 
exists in the vicinity of the pixels. Accordingly, pixel values 
may be characteristic amounts that represent likelihood 
of being red eyes. Here, an example will be described, 
in which pixel values are defined as the characteristic 

55 amounts. 

[0060] The score is an index that represents how likely 
red eyes exist. Correlations among scores and charac- 
teristic amounts are defined. In the case of the above 
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example, pixel values, which are perceived to be red by 
all viewers, are assigned high scores, while pixel values, 
which may be perceived to be red by some viewers and 
brown by other viewers, are assigned lower scores. 
Meanwhile, pixel values that represent colors which are 
clearly not red (for example, yellow) are assigned scores 
of zero or negative scores. Figure 4 is a score table that 
illustrates an example of the correspondent relationship 
between characteristic amounts and scores. 
[0061] Whether the image within the judgment target 
region 8 represents red eyes is judged in the following 
manner. First, characteristic amounts are calculated for 
each pixel within the judgment target region 8. Then, the 
calculated characteristic amounts are converted to 
scores, based on definitions such as those exemplified 
in the score table of Figure 4. Next, the scores of all of 
the pixels within the judgment target region 8 are totaled. 
If the total value of the scores is greater than or equal to 
the threshold value, the subject of the image within the 
judgment target region is judged to be a red eye. If the 
total value of the scores is less than the threshold value, 
it is judged that the image does not represent a red eye. 
[0062] As is clear from the above description, the ac- 
curacy of judgment in the above process depends greatly 
on the definitions of the characteristic amounts, the score 
table, and the threshold value. For this reason, the red 
eye detecting apparatus of the present embodiment per- 
forms learning, employing sample images of red eyes 
and sample images of other subjects (all of which are 1 0 
pixels x 10 pixels). Appropriate characteristic amounts, 
score tables, and threshold values, which are learned by 
the learning process, are employed in judgment. 
[0063] Various known learning methods, such as a 
neural network method, which is known as a machine 
learning technique, and a boosting method, may be em- 
ployed. Images, in which red eyes are difficult to detect, 
are also included in the sample images utilized in the 
learning process. 

For example, the sample images utilized in the learning 
process may include: standard sample images, as illus- 
trated in Figure 5A; images in which the size of the pupil 
is smaller than that of standard sample images, as illus- 
trated in Figure 5B; images in which the center position 
of the pupil is misaligned, as illustrated in Figure 5C; and 
images of incomplete red eyes, in which only a portion 
of the pupil is red, as illustrated in Figures 5D and 5E. 
[0064] The sample images are utilized in the learning 
process, and effective characteristic amounts are select- 
ed from among a plurality of characteristic amount can- 
didates. The judgment process described above is re- 
peated, employing the selected characteristic amounts 
and score tables generated therefor. The threshold value 
is determined so that a predetermined percentage of cor- 
rect judgments is maintained during the repeated judg- 
ments. 

[0065] At this time, the red eye detecting apparatus of 
the present embodiment performs N types of judgment 
(N is an integer greater than or equal to 2) on individual 



judgmenttarget regions, utilizing N types of characteristic 
amounts, score tables, and threshold values. The coor- 
dinates of judgment target regions are registered in a red 
eye candidate list only in cases in which all of the N judg- 
5 ments judge that red eye is present. That is, the accuracy 
of judgment is improved by combining the plurality of 
types of characteristic amounts, score tables, and thresh- 
old values, and only reliable judgment results are regis- 
tered in the list. Note that here, "registered in a red eye 
10 candidate list" refers to recording positional coordinate 
data and the like in the memory medium. 
[0066] Figure 6 is a flow chart that illustrates the N 
types of judgment processes. As illustrated in Figure 6, 
the red eye detecting apparatus first performs a first judg- 
es ment on a set judgment target region, referring to a first 
type of characteristic amount calculating parameters, 
score table and threshold value. The characteristic 
amount calculating parameters are parameters, such as 
coefficients, that define a calculation formula for charac- 
teristic amounts. 

[0067] In the case that the first red eye judgment proc- 
ess judges that red eye exists, the same judgment target 
region is subjected to a second judgment, referring to a 
second type of characteristic amount calculating param- 
eters, score table, and threshold value. In the case that 
the first red eye judgment process judges that red eye is 
not present, it is determined at that point that the image 
within the judgment target region does not represent red 
eye, and a next judgment target region is set. 
[0068] Thereafter, in cases that red eye is judged to 
exist by a (K-1) th judgment process (2<K<N), the same 
judgment target region is subjected to a K th judgment 
process, referring to a type of characteristic amount 
calculating parameters, score table, and thresholdvalue. 
In cases that a (K-1 ) th judgment process judges that red 
eye is not present, then judgment processes for that judg- 
ment target region are ceased at that point. 
[0069] Note that at each judgment, characteristic 
amounts are calculated for each pixel (step S101), the 
characteristic amounts are converted to scores (step 
S1 02), and the scores of all of the pixels within the judg- 
ment target region are totaled (step S103). If the total 
value of the scores is greater than or equal to the thresh- 
old value, the subject of the image within the judgment 
target region is judged to be a red eye; and if the total 
value of the scores is less than the threshold value, it is 
judged that the image does not represent a red eye (step 
S104). 

[0070] The red eye detecting apparatus registers co- 
ordinates of judgment target regions in a red eye candi- 
date list, only in cases in which an N-th judgment, which 
refers to an N-th type of characteristic amount calculating 
parameter, score table, and threshold value, judges that 
red eye is present. 

[0071] In the judgment process described above, it is 
assumed that red portions included in the image 1 are of 
sizes thatfit within a 10 pixel x 10 pixel region. In actuality, 
however, there are cases in which a red eye 7d included 
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in the image 1 is larger than the 1 0 pixel x 1 0 pixel judg- 
ment target region 8, as illustrated in Figure 7A. For this 
reason, the red eye detecting apparatus of the present 
embodiment performs the aforementioned judgment 
processes not only on the image 1 input thereto, but on 
a low resolution image 13, generated by reducing the 
resolution of the image 1 , as well. 
[0072] As illustrated in Figure 7B, if the resolution of 
the image 1 is reduced, the red eye 7d fits within the 10 
pixel x 1 0 pixel judgment target region 8. It becomes pos- 
sible to perform judgments on the low resolution image 
13 employing the same characteristic amounts and the 
like as those which were used in the judgments per- 
formed on the image 1. The image having a different 
resolution may be generated at the point in time at which 
the image 1 is input to the red eye detecting apparatus. 
Alternatively, resolution conversion maybe administered 
on the image 1 as necessary during execution of the red 
eye candidate detecting step. 

[0073] Note that judgments may be performed by mov- 
ing the judgment target region 8 in small increments (for 
example, 1 pixel) . In these cases, a single red eye may 
be redundantly detected by judgment processes for dif- 
ferent judgment target regions 9 and 1 0, as illustrated in 
Figure 8. The single red eye may be registered in the red 
eye candidate list as separate red eye candidates 1 1 and 
12. There are also cases in which a single red eye is 
redundantly detected during detecting processes admin- 
istered on images having different resolutions. 
[0074] For this reason, the red eye detecting apparatus 
of the present embodiment confirms the coordinate in- 
formation registered in the red eye candidate list after 
scanning of the judgment target region is completed for 
all images having different resolutions. In cases that a 
plurality of pieces of coordinate information that clearly 
represent the same red eye are found, only one piece of 
the coordinate information is kept, and the other pieces 
are deleted from the list. Specifically, the piece of coor- 
dinate information that represents the judgment target 
region having the highest score total is kept as a red eye 
candidate, and the other candidates are deleted from the 
list. 

[0075] The red eye candidate list, which has been or- 
ganized as described above, is output as processing re- 
sults of the red eye candidate detecting step 2, and uti- 
lized in the following face detecting step 3. 
[0076] In the red eye candidate detecting step of the 
present embodiment, processing time is reduced without 
decreasing the accuracy of detection. This is accom- 
plished by adjusting the resolution of images employed 
in the detection, the manner in which the judgment target 
regions are set, and the order in which the N types of 
characteristic amount calculating parameters are uti- 
lized. Hereinafter, methods for improving the processing 
efficiency of the red eye candidate detecting step will be 
described further. [Methods for Improving Red Eye Can- 
didate Detection Efficiency] 

[0077] The methods for improving the efficiency of the 



red eye candidate detecting step described below may 
be employed either singly or in combinations with each 
other. 

[0078] A first method is a method in which character- 

5 istic amounts are defined such that the amount of calcu- 
lations is reduced for judgments which are performed 
earlier, during the N types of judgment. As has been de- 
scribed with reference to Figure 6, the red eye detecting 
apparatus of the present embodiment does not perform 

10 (K+1) th judgment processes in cases in which the K th 
judgment process judges that red eye is not present. This 
means that judgment processes, which are performed at 
earlier stages, are performed more often. Accordingly, 
by causing the processes which are performed often to 

15 be those that involve small amounts of calculations, the 
efficiency of the entire process can be improved. 
[0079] The definition of the characteristic amounts de- 
scribed above, in which the characteristic amounts are 
defined as the values of pixels (x, y), is the example that 

20 involves the least amount of calculations. 

[0080] Another example of characteristic amounts 
which may be obtained with small amounts of calcula- 
tions is differences between pixel values (x, y) and pixel 
values (x+dx, y+dy) . The differences between pixel val- 

25 ues may serve as characteristic amounts that represent 
likelihood of being red eyes, because colors in the pe- 
riphery of pupils are specific, such as white (whites of the 
eyes) or skin color (eyelids). Similarly, combinations of 
differences between pixel values (x, y) and pixel values 

30 (x+dx1 , y+dy1 ) and differences between pixel values (x, 
y) and pixel values (x+dx2, y+dy2) may also serve as 
characteristic amounts that represent likelihood of being 
red eyes. Combinations of differences among four or 
more pixel values may serve as characteristic amounts. 

35 Note that values, such as dx, dx1 , dx2, dy, dy1 , and dy2, 
which are necessary to calculate the characteristic 
amounts, are recorded as characteristic amount calcu- 
lating parameters. 

[0081] As an example of characteristic amounts that 

40 require more calculations, averages of pixel values within 
a 3 x 3 pixel space that includes a pixel (x, y) may be 
considered. Combinations of differences among pixel 
values in the vertical direction and the horizontal direc- 
tion, within a 3 x 3 pixel space having a pixel (x, y) at its 

45 center, may also serve as characteristic amounts. The 
difference among pixel values in the vertical direction 
may be obtained by calculating weighted averages of the 
3x3 pixels, employing a filter such as that illustrated in 
Figure 9A. Similarly, the difference among pixel values 

50 in the horizontal direction may be obtained by calculating 
weighted averages of the 3 x 3 pixels, employing a filter 
such as that illustrated in Figure 9B. As examples of char- 
acteristic amounts that involve a similar amount of cal- 
culations, there are: integral values of pixels which are 

55 arranged in a specific direction; and average values of 
pixels which are arranged in a specific direction. 
[0082] There are characteristic amounts that require 
even more calculations. Gradient directions of pixels (x, 
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y), that is, the directions in which the pixel value (color 
density) changes, may be obtained from values calculat- 
ed by employing the filters of Figures 9A and 9B. The 
gradient directions may also serve as characteristic 
amounts that represent likelihood of being red eyes. The 
gradient direction may be calculated as an angle 9 with 
respect to a predetermined direction (for example, the 
direction from a pixel (x, y) to a pixel (x+dx, y+dy)). In 
addition, "Detection Method of Malignant Tumors in DR 
Images- Iris Filter-", Kazuo Matsumoto et al., Journal of 
the Electronic Information Communication Society, Vol. 
J75-D11, No. 3, pp. 663-670, 1992 discloses a method 
by which images are evaluated based on distributions of 
gradient vectors. Distributions of gradient vectors may 
also serve as characteristic amounts that represent like- 
lihood of being red eyes. 

[0083] A second method is based on the same princi- 
ple as the first method. The second method classifies 
characteristic amounts in to two groups. One group in- 
cludes characteristic amounts that require relatively 
small amounts of calculations, and the other group in- 
cludes characteristic amounts that require large amounts 
of calculations. Judgment is performed in steps. That is, 
the judgment target region is scanned on the image twice. 
[0084] Figure 1 0 is a flow chart that illustrates the judg- 
ment process in the case that the second method is em- 
ployed. As illustrated in the flow chart, during the first 
scanning, first, the judgment target region is set (step 
S201 ). Then, judgment is performed on the judgmenttar- 
get region employing only the characteristic amounts that 
require small amounts of calculations (step S202) . The 
judgment target region is moved one pixel at a time and 
judgment is repeated, until the entirety of the image is 
scanned (step S203) . During the second scanning, judg- 
ment target regions are set at the peripheries of the red 
eye candidates detected by the first scanning (step 
S204) . Then, judgment is performed employing the char- 
acteristic amounts that require large amounts of calcu- 
lations (step S205). Judgment is repeated until there are 
no more red eye candidates left to process (step S207). 
[0085] In the second method, the judgment processes 
employing the characteristic amounts that require large 
amounts of calculations are executed on a limited 
number of judgment target regions. Therefore, the 
amount of calculations can be reduced as a whole, there- 
by improving processing efficiency. In addition, in the 
second method, the judgment results obtained by the 
first scanning may be output to a screen or the like prior 
to performing the second detailed judgment. That is, the 
amount of calculations in the first method and in the sec- 
ond method is substantially the same. However, it is pref- 
erable to employ the second method, from the viewpoint 
of users who observe reaction times of the red eye de- 
tecting apparatus. 

[0086] Note that the number of groups that the char- 
acteristic amounts are classified in according to the 
amount of calculations thereof is not limited to two 
groups. The characteristic amounts may be classified in- 



to three or more groups, and the judgment accuracy may 
be improved in a stepwise manner (increasing the 
amount of calculations). In addition, the number of char- 
acteristic amounts belonging to a single group may be 

5 one type, or a plurality of types. 

[0087] A third method is a method wherein the judg- 
ment target region is moved two or more pixels at a time 
during scanning thereof, as illustrated in Figure 11, in- 
stead of one pixel at a time. Figure 1 1 illustrates an ex- 

10 ample in which the.judgment target region is moved in 
10 pixel increments. If the total numberof judgment target 
regions decreases, the amount of calculations as a whole 
is reduced, and therefore processing efficiency can be 
improved. Note that in the case that the third method is 

15 employed, it is preferable that learning is performed using 
a great number of sample images, in which the centers 
of red eyes are misaligned, such as that illustrated in 
Figure 5C. 

[0088] A fourth method is a method wherein judgment 
20 processes are performed on a lower resolution image 
first. Judgment target regions are relatively larger with 
respect to lower resolution images as compared to higher 
resolution images. Therefore, larger portions of the im- 
age can be processed at once. Accordingly, judgment is 
25 performed on a lower resolution image first, and regions 
in which red eyes are clearly not included are eliminated. 
Then, judgment is performed on a higher resolution im- 
age only at portions that were not eliminated by the first 
judgment. 

30 [0089] The fourth method is particularly effective for 
images in which people with red eyes are pictured at the 
lower halves thereof, and dark nightscapes are pictured 
at the upper halves thereof. Figure 12A and Figure 12B 
illustrate an example of such an image. Figure 1 2A illus- 

35 trates a low resolution image 13, and Figure 12B illus- 
trates a high resolution image 1 , which was input to the 
red eye detecting apparatus. 

[0090] As is clear from Figure 12A and Figure 12B, if 
the judgment target region 8 is scanned over the entirety 

40 of the low resolution image 1 3 first, the upper half of the 
image that does not include red eyes can be eliminated 
as red eye candidates by a process that involves small 
amounts of calculations. Therefore, the judgment target 
region 8 is scanned over the entirety of the low resolution 

45 image 1 3, and red eye candidates are detected. Then, a 
second candidate detection process is performed on the 
image 1 , only in the peripheries of the detected red eye 
candidates. Thereby, the number of judgments can be 
greatly reduced. Note that in the case that this method 

50 is employed, it is preferable that learning is performed 
using a great number of sample images, in which the red 
eyes are small, such as that illustrated in Figure 5B. 
[0091] Next, a fifth method, which is effective if used 
in combination with the third or the fourth method, will be 

55 described with reference to Figure 13. The third and 
fourth methods are capable of quickly narrowing down 
red eye candidates with small amounts of calculations. 
However, the detection accuracy of the positions of the 
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detected red eye candidates is not high. Therefore, the 
fifth method searches for red eye candidates in the vicin- 
ities of the narrowed down red eye candidates. In the 
case that the fourth method is employed, the search for 
red eye candidates in the vicinities of the red eye candi- 
dates is performed on the higher resolution image. 
[0092] For example, consider a case in which a red 
eye candidate having a pixel 14 at its center is detected 
by the third or fourth method. In this case, a judgment 
target region 1 5 is set so that the pixel 1 4 is at the center 
thereof. Then, judgment is performed employing the 
same characteristic amounts, score table, and threshold 
value as the previous judgment, or by employing char- 
acteristic amounts, score table, and threshold value hav- 
ing higher accuracy. Further, a highly accurate judgment 
is also performed within a judgment target region 1 7, hav- 
ing a pixel 16, which is adjacent to the pixel 14, at the 
center thereof. 

[0093] In a similar manner, judgment target regions 
are set having the other 7 pixels adjacent to the pixel 14 
at the centers thereof, and judgments regarding whether 
red eye exists therein are performed. Alternatively, judg- 
ment may be performed on the 16 pixels that are ar- 
ranged so as to surround the 8 pixels adjacent to the 
pixel 1 4. As a further alternative, a plurality of judgment 
target regions that overlap at least a portion of the judg- 
ment target region 15 may be set, and judgment per- 
formed thereon. 

[0094] In the case that a different red eye candidate is 
detected during the search of the peripheral region of the 
red eye candidate, the coordinates of the different red 
eye candidate (for example, the coordinates of the pixel 
16) are added to the list. By searching the peripheral 
region of the red eye candidate in detail, the accurate 
position of the red eye candidate may be obtained. 
[0095] Note that in this case, a single red eye is redun- 
dantly detected. Therefore, the aforementioned organi- 
zation is performed after searching is complete. Specif- 
ically, coordinates of the judgment target region having 
the highest score total, from among the coordinates 
which have been judged to be red eyes and added to the 
list, is kept as a red eye candidate, and the other coor- 
dinates are deleted from the list. 

[0096] Note that in the fifth method, the accuracy of 
judgment is improved over the previous judgment when 
searching for redeye candidates within the narrowed 
down regions. Thereby, the positional accuracy of the 
detected red eye candidates is improved. A sixth method, 
to be described below, is applicable to cases in which 
the judgment accuracy of the second and following judg- 
ments is desired to be improved over that of previous 
judgments. 

[0097] In the sixth method, characteristic amounts are 
classified into two groups, in the same manner as in the 
second method. One group includes characteristic 
amounts that require relatively small amounts of calcu- 
lations, and the other group includes characteristic 
amounts that require large amounts of calculations. 



[0098] Figure 1 4 is a flow chart that illustrates the judg- 
ment process in the case that the sixth method is em- 
ployed. As illustrated in the flow chart, during the first 
scanning, first, the judgment target region is set (step 

5 S201). Then, judgment is performed on the judgment tar- 
get region employing only the characteristic amounts that 
require small amounts of calculations (step S202) . The 
judgment target region is moved two pixels at a time as 
described in the third method, and judgment is repeated 

10 until the entirety of the image is scanned (step S203). 
Alternatively, the first scanning may be performed on a 
lower resolution image, as described in the fourth meth- 
od. 

[0099] During the second scanning, judgment target 
15 regions are set in the peripheries of the red eye candi- 
dates, which have been detected by the first scanning, 
as described in the fifth method (step S204). Then, judg- 
ments are performed (step S206) until there are no more 
red eye candidates left to process (step S207). Both char- 
ge acteristic amounts that require small amounts of calcu- 
lations and those that require large amounts of calcula- 
tions are employed during the judgments of step S206. 
However, during the judgment of step S206 employing 
the characteristic amounts that require small amounts of 
25 calculations, the threshold values are set higher than dur- 
ing the judgment of step S202. Specifically, the threshold 
value is set low during the judgment of step S202, to 
enable detection of red eyes which are located at posi- 
tions off center within the judgment target regions. On 
30 the other hand, the judgment of step 206 sets the thresh- 
old value high, so that only red eyes, which are positioned 
at the centers of the judgment target regions, are detect- 
ed. Thereby, the positional accuracy of the red eyes de- 
tected in step S206 is improved. 
35 [0100] Note that the number of groups that the char- 
acteristic amounts are classified in according to the 
amount of calculations thereof is not limited to two 
groups. The characteristic amounts may be classified in- 
to three or more groups, and the judgment accuracy may 
40 be improved in a stepwise manner (increasing the 
amount of calculations). In addition, the number of char- 
acteristic amounts belonging to a single group may be 
one type, or a plurality of types. 

[0101] The red eye detecting apparatus of the present 
45 embodiment employs the above methods either singly 
or in combination during detection of red eye candidates. 
Therefore, red eye candidates may be detected efficient- 
ly. 

[01 02] Note it would be clear to those skilled in the art 
50 that the methods for improving efficiency described 
above are not limited to cases in which detection is per- 
formed by the procedures of detecting red eye candi- 
dates, detecting faces, and confirming red eyes. The 
above methods may be applied to any other detecting 
55 process that involves calculations of characteristic 
amounts. 
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[Face Detecting Step] 

[01 03] Next, the face detecting step 3 will be described. 
There are various known face detecting methods, but 
each method has different detection properties. Here, 
"detection properties" refer to the kinds of detection each 
method is superior for. Examples of detection properties 
are: "fast detection"; "high detection rates"; "high detec- 
tion accuracy"; "superior in detection of common faces 
(for example, forward facing faces without facial hair or 
glasses)" (hereinafter, referred to as "common face 
priority") ; and "superior in detection of special faces (for 
example, faces in profile, faces with facial hair, and faces 
with glasses)" (hereinafter, referred to as "special face 
priority"). In the present specification, "high detection 
rates" refers to low incidences of misdetections (faces 
that are not detected as faces), and "high detection ac- 
curacy" refers to low incidences of false positive detec- 
tions (objects being detected as faces, even though they 
are not faces). 

[0104] Note that the detection properties do not de- 
pend solely on detection algorithms, and also depend on 
other conditions during execution of the detection algo- 
rithms. For example, even if the algorithm is the same in 
that a predetermined process is executed while scanning 
(shifting) a judgment target region on an image, the de- 
tection speed will differ according to the scanning density 
of the judgment target region. The same principle applies 
to the detection rate and the detection accuracy. 
[0105] The face detecting step 3 may select a single 
face detecting method from among a plurality of face de- 
tecting methods, each having different detection proper- 
ties, when executing the face detecting process. Howev- 
er, in the present embodiment, a plurality of different face 
detecting methods are employed in combination as re- 
quired, in order to decrease misdetection of faces. This 
is because even if a face, which is undetectable by a 
certain face detecting method, is included in an image, 
the face can be positively detected by employing combi- 
nations of a plurality of face detection methods. 
[01 06] However, it is not preferable for processing time 
to increase, even if misdetections are decreased. It is 
necessary to suppress increases in processing time, 
while accurately and positively detecting faces. There- 
fore, the present embodiment applies the following rules 
when combining a plurality of face detecting methods 
having different detection properties. 

Rule 1: Face detecting methods having high detec- 
tion speeds are employed before face detecting 
methods having low detection speeds. 

Rule 2: Face detecting methods having high detec- 
tion accuracies are employed before face detecting 
methods having low detection accuracies. 

Rule 3: Common face priority face detecting meth- 
ods are employed before special face priority face 



detecting methods. 

Rule 4: Face detecting methods are employed se- 
quentially to detect faces, but if a face is detected by 
5 a certain face detecting method, subsequent detec- 

tions by other face detecting methods are not per- 
formed. 

[0107] In the present embodiment, the face detecting 

10 step 3 is performed by combining a first, a second, and 
a third face detecting method, to be described later, ac- 
cording to the above rules. The first and second face 
detecting methods have the same detection algorithm, 
but have different detection properties. The first face de- 

15 tecting method is superior in detection of forward facing 
faces, and the second face detecting method is superior 
in detection of faces in profile. The third face detecting 
method has an algorithm different from that of the first 
and second face detecting methods. 

20 [0108] The face detecting algorithms of the first face 
detecting method and the second face detecting method 
are basically the same as the eye detecting algorithm of 
the red eye candidate detecting step 2. Specifically, the 
algorithms are similar in that: learning employing sample 

25 images is performed in advance, to select appropriate 
characteristic amounts, score tables and the like; optimal 
threshold values are set based on the learning; charac- 
teristic amounts are calculated for each pixel within judg- 
ment target regions, converted to scores, the scores are 

30 totaled and compared against the threshold values; 
searching is performed while varying the resolution of 
the image; and setting judgment target regions within the 
image during searches to investigate how many charac- 
teristics inherent to faces are present in the images within 

35 the judgment target regions. 

[0109] However, the samples employed for learning, 
to select appropriate characteristic amounts, score ta- 
bles and the like, are samples of faces, and not eyes. In 
the first face detecting method, learning is performed with 

40 emphasis placed on forward facing faces. For example, 
many images of front facing faces are included in the 
sample images employed for learning. Therefore, the first 
face detecting method is superior in detecting front facing 
faces. On the other hand, in the second face detecting 

45 method, learning is performed with emphasis placed on 
faces in profile. Therefore, the second face detecting 
method is superior in detecting faces in profile. 
[01 1 0] The first and second face detecting methods do 
not search for faces within the entirety of the image. In- 

50 stead, the first and second face detecting methods em- 
ploy the red eye candidates, detected by the red eye 
candidate detecting step 2, as reference points. That is, 
faces are searched for only in the peripheries of the red 
eye candidates. Figure 15 illustrates a state in which a 

55 judgment target region 20 is set on an image 1 , in which 
red eye candidates 18 and 19 have been detected. 
[0111] In addition, in the first and second detecting 
methods, scanning of the judgment target region 20 is 
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not limited to horizontal movement in the vicinities of the 
red eye candidates, as illustrated in Figure 15. Searching 
is also performed while rotating the judgment target re- 
gion 20, as illustrated in Figure 16. This is because the 
values of characteristic amounts for faces vary greatly 5 
depending on the orientation of the face, unlike those for 
eyes (pupils). In the present embodiment, if faces are not 
detected with the judgment target region in a certain ori- 
entation, the judgment target region is rotated 30 de- 
grees. Then, characteristic amounts are calculated, the 10 
characteristic amounts are converted to scores, and the 
totaled scores are compared against the threshold val- 
ues, within the rotated judgment target region. 
[0112] The first and second face detecting methods 
judge whether faces exist within the judgment target re- 15 
gion based on characteristic amounts, which are extract- 
ed by wavelet conversion. Figure 17 is a flow chart that 
illustrates the face detecting process. 
[01 13] The red eye detecting apparatus first adminis- 
ters wavelet conversion on Y (luminance) components 20 
of the image within the judgment target region (step 
S301). Thereby, a 1/4 size sub band image, an LLO im- 
age, an LHO image, an HLO image, and an HHO image 
(hereinafter, these will be collectively be referred to as 
"level 0 images") are generated. In addition, a 1/16 size 25 
sub band image, an LL1 image, an LH1 image, an HL1 
image, and an HH1 image (hereinafter, these will be col- 
lectively be referred to as "level 1 images") are generated. 
Further, a 1/64 size sub band image, an LL2 image, an 
LH2 image, an HL2 image, and an HH2 image (herein- 30 
after, these will be collectively referred to as "level 2 im- 
ages") are generated. 

[0114] Thereafter, the red eye detecting apparatus 
employs local scattering to normalize and quantize the 
sub band images, which have been obtained by wavelet 35 
conversion (step S302). 

[01 15] In the case that images are analyzed by wavelet 
conversion, LH images are obtained, in which the edges 
in the horizontal direction are emphasized. Further, HL 
images are obtained, in which the edges in the vertical 40 
direction are emphasized. For this reason, characteristic 
amounts are calculated from within level 0, level 1, and 
level 2 LH and HL images (step S303) during a face judg- 
ing process, as illustrated in Figure 18. In the present 
embodiment, arbitrary four point combinations of the 45 
wavelet coefficients of the LH images and the HL images 
are defined as characteristic amounts that represent like- 
lihood of being faces. Next, the calculated characteristic 
amounts are converted to scores (step S304), the scores 
are totaled (step S305), and the total scores are com- 50 
pared against threshold values (step S306), in a manner 
similar to that of the red eye candidate detecting step 2. 
The red eye detecting apparatus judges the image within 
the judgment target region to be a face if the total score 
is greaterthan or equal to the threshold value, and judges 55 
that the image is not of a face if the total score is less 
than the threshold value. Note that the above processes 
utilize the fact that edges exist at portions that correspond 



to the outlines of faces, within images that represent fac- 
es, and at the boundaries between eyes and skin. There- 
fore, edges, which are included at predetermined por- 
tions of images, are defined as characteristic amounts 
that represent likelihood of being faces, to perform judg- 
ment and detection. That is, face detection is performed 
based only on data related to structures. An example of 
a method other than that which employs wavelet coeffi- 
cients is that in which differences between the pixel val- 
ues of pixels (x, y) and pixels (x+dx1 , y+dy1 ) are defined 
as characteristic amounts. 

[0116] Alternatively, processes that utilize edge data 
and color data in combination may be considered. That 
is, the characteristic amounts that represent likelihood of 
being faces are not necessarily limited to those that are 
calculated based on the Y components of images. The 
characteristic amounts may be those which are calculat- 
ed based on the Y components, the Cb components, and 
the Cr components. In the case that judgment is per- 
formed based only on characteristic amounts related to 
structures, such as edges, the possibility that structures 
similar to faces will be detected as false positives cannot 
be completely eliminated. However, false positive detec- 
tions can be prevented, by utilizing the color data in com- 
bination with the edge data. 

[0117] The third face detecting method detects skin 
colored regions in the vicinities of the red eye candidates, 
and judges the skin colored regions as faces. Because 
this method employs only color data to judge faces, the 
detection accuracy is low compared to the first and sec- 
ond face detecting methods. However, the detection rate 
of the third face detecting method is high. In the case that 
a face is detected by the third face detecting method, the 
face is correlated with red eye candidates and registered 
in the face list. 

[0118] In the face detecting step 3, first, a first face 
detecting process is performed employing the first face 
detecting method (first subject detecting process). If a 
face is detected, then the process ends. If a face is not 
detected, then a second face detecting process is per- 
formed employing the second face detecting method 
(second subject detecting process). If a face is detected 
by the second face detecting process, the process ends. 
If a face is not detected by the second face detecting 
process, a third face detecting process is performed em- 
ploying the third face detecting method. If a face is de- 
tected by the third face detecting process, the process 
ends. If a face is not detected by the third face detecting 
process, detection results indicating that a face was not 
detected is output, and the face detecting step 3 ends. 
[01 19] In the case that a face is detected by the afore- 
mentioned search, the red eye detecting apparatus reg- 
isters the face in a face list, correlated with the red eye 
candidate that served as the reference point for the 
search. In the example illustrated in Figure 1 5 and Figure 
1 6, the red eye 1 8 and a face 21 are correlated and reg- 
istered in the face list. In addition, the red eye 1 9 and the 
face 21 are correlated and registered in the face list. 
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[0120] In the case that the same face is redundantly 
detected, the registered information is organized. In the 
aforementioned example, information regarding the face 
21, the red eye candidates 18 and 19 are consolidated 
into one piece of information. The consolidated informa- 
tion is reregistered in the face list. The face list is referred 
to in the red eye confirming step 4, to be described below. 

[Methods for Improving Face Detection Efficiency] 

[0121] It would be clear to those skilled in the art that 
the methods to improve the efficiency of red eye candi- 
date detection described previously can be applied to the 
first or second face detecting methods, to improve the 
efficiency of the face detecting step. 
[0122] For example, the amount of calculations re- 
quired differs between the characteristic amounts, which 
are defined by the differences between pixel values of 
pixels (x, y) and pixels (x+dx1 , y+dy1 ), and those which 
are defined by the gradient directions of edges at pixels 
(x, y). Accordingly, in the case that a plurality of types of 
characteristic amounts are defined as those that repre- 
sent likelihood of being faces, the characteristic amounts 
may be grouped according to the amount of calculations 
required therefor. The amount of calculations can be re- 
duced, without deteriorating the detection accuracy, by 
performing judgments in order, by employing character- 
istic amounts that require less amounts of calculation 
first. 

[0123] In addition, by shifting the judgment target re- 
gions by greater increments of pixels during scanning, 
the processing time can be reduced. Further, lower res- 
olution images may be employed to perform judgments, 
based on characteristic amounts that require less 
amounts of calculations, to eliminate regions that do not 
include faces first. Thereafter, higher resolution images 
may be employed to perform judgments, based on char- 
acteristic amounts that require greater amounts of cal- 
culations. Thereby, the amount of calculations and the 
processing time can be reduced. Still further, after faces 
are detected in a judgment target region, a plurality of 
judgment target regions may be set such that at least a 
portion thereof overlaps with the judgment target region, 
and similar judgments may be performed with respect to 
the plurality of judgment target regions. Alternatively, 
judgments may be performed using the same character- 
istic amounts, but with a higher threshold value. Thereby, 
the positional accuracy in detection can be improved. 
[0124] Note that each of the above methods for im- 
proving efficiency may be employed either singly or in 
combination, in the same manner as in the red eye can- 
didate detecting process. 

[Red Eye Confirming Step] 

[0125] Next, the red eye confirming step 4 will be de- 
scribed. The red eye confirming step 4 judges whether 
the red eye candidates, which have been correlated with 



faces and recorded in the face detecting step 3, are true 
red eyes. In other words, the red eye confirming step 4 
verifies the detection results of the red eye candidate 
detecting step 2. The judgment of red eye in the red eye 

5 confirming step 4 is the final judgment that affects the 
detection results of the red eye detecting apparatus. 
Therefore, it is necessary that the judgment of red eye 
to be performed more accurately than that performed in 
the red eye candidate detecting step 2. Hereinafter, the 

10 red eye judgment process performed by the red eye con- 
firming step 4 will be described. 

[0126] Figure 19 illustrates the red eye candidates 18 
and 19, which have been detected from the image 1 by 
the red eye candidate detecting step 2, the face 21 , which 

15 has been detected by the face detecting step 3, and 
search regions 22, which have been set in the image 1 
by in the red eye confirming step 4. The objective of the 
red eye candidate detecting step 2 is to detect red eye 
candidates. Therefore, the search region for the red eye 

20 candidate detecting step 2 was the entirety of the image. 
In contrast, the objective of the red eye confirming step 
4 is to verify the detection results of the red eye candidate 
detecting step 2. Therefore, the search region may be 
limited to the vicinities of the red eye candidates, as il- 

25 lustrated in Figure 19. 

[0127] During the red eye confirming step 4, the red 
eye detecting apparatus refers to information regarding 
the size and orientation of faces, obtained in the face 
detecting step 3. Thereby, the orientations of the red eye 

30 candidates are estimated, and the search regions are set 
according to the sizes and orientations of the red eye 
candidates. That is, the search regions are set so that 
the vertical directions of the pupils match the vertical di- 
rections of the search regions. In the example illustrated 

35 in Figure 1 9, the search regions 22 are inclined to match 
the inclination of the face 21 . 

[01 28] Next, the red eye judgment process performed 
within the search regions 22 will be described. Figure 20 
illustrates the search region 22 in the vicinity of the red 
40 eye candidate 18. In the red eye judgment process, a 
judgment target regions 23 are set within the search re- 
gion 22. 

[01 29] Next, characteristic amounts are calculated for 
each pixel within the judgment target region 23, and the 

45 calculated characteristic amounts are converted to 
scores that represent likelihood of being red eyes by em- 
ploying a score table, in the same manner as in the red 
eye candidate detecting step. Then, the red eye candi- 
dates are judged to be red eyes if the total value of the 

50 scores corresponding to each pixel within the judgment 
target region exceeds a threshold value. The red eye 
candidates are judged not to be red eyes if the total value 
of the scores is less than the threshold value. 
[0130] The judgmenttarget region 23 is scanned within 

55 the search region 22, and the judgment described above 
is performed repeatedly. In the case of the red eye con- 
firming step 4, red eye candidates are present within the 
search region 22, as opposed to the red eye candidate 
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detecting step 2. Accordingly, in the case that judgments 
are performed by scanning the judgment target region 
23 within the search region 22, many judgment results 
indicating red eye should be obtained. There are cases 
in which the number of positive judgments indicating red 
eye is small, regardless of the fact that the judgments 
were performed by scanning the judgment target region 
23 within the search region 22. In these cases, there is 
a possibility that the red eye candidate 18 is not a true 
red eye. This means that the number of times that red 
eye is judged to exist, during scanning of the judgment 
target region 23, is an effective index that represents the 
reliability of the detection results of the red eye candidate 
detecting step 2. 

[0131] A plurality of images having different resolu- 
tions are employed during judgment of red eye in the red 
eye confirming step 4, in the same manner as in the red 
eye candidate detecting step 2. Figures 21 A, 21 B, and 
21 C illustrate states in which search regions 22, 25, and 
27, all of the same size, are respectively set in the vicinity 
of the red eye candidate 1 8, within images 1 , 24, and 26, 
which are of different resolutions. 
[0132] The resolutions of images are finely varied in 
the red eye confirming step 4, unlike in the red eye can- 
didate detecting step 2. Specifically, the resolution is 
changed so that the image 24 of Figure 21 B has about 
98% of the number of pixels of the image 1 of Figure 21 A, 
and so that the image 26 of Figure 21 C has about 96% 
of the number of pixels of the image 1 of Figure 21 A. 
[0133] In the red eye confirming step 4 of the present 
embodiment, the number of times that red eye was 
judged to exist in the images having different resolutions 
are totaled. If this total number is greater than a prede- 
termined number, it is judged that the red eye candidate 
is a true red eye. On the other hand, if the total number 
is the predetermined number or less, it is judged that the 
red eye candidate was a false positive detection, and that 
it is not a true red eye. In this case, the red eye detecting 
apparatus deletes information regarding the red eye can- 
didate from every list that it is registered in. Alternatively, 
the total values of the scores may be utilized to judge 
whether the red eye candidates are true red eyes, instead 
of the total number of judgments. As a further alternative, 
both the total number of judgments and the total values 
of the scores may be utilized to judge whether the red 
eye candidates are true red eyes. 
[0134] In the case that red eye candidates are judged 
to be true red eyes, the red eye confirming step 4 then 
confirms the positions of the red eyes. As described 
above, if judgments are performed by scanning the judg- 
ment target region within the search regions, positive 
judgments are obtained atmanyjudgmenttarget regions. 
Therefore, the red eye detecting apparatus of the present 
invention defines a weighted average of the center co- 
ordinatesofthejudgmenttarget regions, in which positive 
judgments were obtained, as the value that represents 
the position of the red eye. The weighting is performed 
corresponding to the total score, which was obtained dur- 



ing judgment, of the judgment target regions. 
[0135] Figure22 isadiagramforexplainingthe method 
by which the positional coordinates of red eyes are con- 
firmed. Figure 22 illustrates the search region 22 and the 

5 center coordinates (indicated by x's) of the judgment tar- 
get regions in which positive judgments were obtained. 
In the example of Figure 22, positive judgments were 
obtained for M (M is an arbitrary integer, in this case, 48) 
judgment target regions. In this case, the position (x, y) 

10 of the red eye is represented by the following formulas: 

i<M 

x = (£Sixi)/M 

15 <=° 



i<M 

y = QTSiyi)/M 

wherein (xi, yi) are the center coordinates of an i-th judg- 
ment target region (0<i<M), and Si is the total score ob- 
tained by the red eye judgment processes in the i-th judg- 
ment target region. 

[0136] Figure 23 is a flow chart that illustrates proc- 
esses of the red eye confirming step. As illustrated in the 
flow chart, the first process in the red eye confirming step 
is the setting of search regions in the vicinities of red eye 
candidates (step S401) . Next, red eye judgment, as has 
been described with reference to Figures 1 9 through 21 , 
is performed within the search ranges (step S402). When 
the searching within the search region is completed (step 
S403), the number of positive judgments is compared 
against the predetermined number (step S404). In the 
case that the number of positive judgments is less than 
or equal to the predetermined number, the red eye can- 
didate is deleted from the list. In the case that the number 
of positive judgments is greater than the predetermined 
number, the position of the red eye is confirmed (step 
S405) by the process described with reference to Figure 
22. The red eye confirming step 4 is completed when the 
above processes are completed for all of the red eye 
candidates detected in the red eye candidate detecting 
step 2. 

[01 37] Note that the characteristic amounts, the score 
tables, and the threshold values, which are employed in 
the red eye confirming step 4 may be the same as those 
which are employed in the red eye candidate detecting 
step 2. Alternatively, different characteristic amounts, 
score tables, and threshold values may be prepared for 
the red eye confirming step 4. 

[0138] In the case that different characteristic 
amounts, score tables, and threshold values are defined 
for the red eye confirming step 4, only images that rep- 
resent standard red eyes are employed as sample im- 
ages during learning. That is, learning is performed using 
only sample images of red eyes having similar sizes and 
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orientations. Thereby, detection is limited to true red 
eyes, and the accuracy of judgment is improved. 
[0139] In the red eye candidate detecting step 2, it is 
preferable that the variation among sample images, 
which are employed during learning, is not decreased, 
because a decrease in variation would lead to red eye 
candidates not being detected. However, the red eye 
confirming step 4 is a process that verifies the detection 
results of the red eye candidate detecting step 2, and 
employs search regions in the vicinities of the detected 
red eye candidates. Therefore, the variation among sam- 
ple images, which are employed during learning, may be 
comparatively small. In the red eye confirming step 4, 
the smaller the variation in sample images, which are 
employed during learning, the stricter the judgment 
standards become. Therefore, the accuracy of judgment 
is improved over that of the red eye candidate detecting 
step 2. 

[01 40] Note that it would be clear to those skilled in the 
art that the methods to improve the efficiency of red eye 
candidate detection described previously can be applied 
to the red eye confirming step 4. In addition, the reliability 
of the detection results is not represented only by the 
total scores or the number of judgments, in the red eye 
confirming step described above. The reliability of the 
detection results is also represented in the size and po- 
sitional relationships between the eyes and faces, when 
the red eye candidates and the faces are correlated with 
each other. Therefore, data regarding the size and posi- 
tional relationships may also be utilized to render final 
judgment regarding whether red eye candidates are true 
red eyes. By verifying the detection results of the red eye 
candidate detecting step 2 in the red eye confirming step 
4, the detection accuracy can be improved, and also the 
accurate positional coordinates of the detected red eyes 
can also be obtained. In the present embodiment, judg- 
ment is repeated while moving thejudgmenttarget region 
in fine increments in the vicinities of the red eye candi- 
dates. In addition, judgment is repeated while the reso- 
lution of the image is finely varied. Then, the positions of 
the red eyes are confirmed by obtaining weighted aver- 
ages, based on the scores. Therefore, even if judgment 
is erroneous in a given region, the effect that this result 
will exert on the final output is small. 
[0141] The method of the present embodiment re- 
quires the three steps of: red eye candidate detection; 
face detection; and red eye confirmation. Therefore, it 
may appear that the number of processes is increased 
compared to conventional methods. However, the 
amount of calculations involved in the red eye confirming 
step 4 is far less than that involved in characteristic ex- 
traction processes administered on faces. In addition, 
because the search regions are limited to the vicinities 
of red eye candidates, neither the amount of processing 
nor the complexity of the apparatus are greatly increased 
compared to conventional methods and apparatuses. 
[01 42] Note that according to the present embodiment, 
first, candidates are detected by performing judgment 



under loose conditions. Then, the detection results are 
verified by performing judgment with stricter conditions 
in the vicinities of the candidates. This method is not lim- 
ited to detecting red eye, but is applicable to detection of 

5 any subject. Verification of detection results according to 
the number of positive judgments of the subject, and con- 
firming the position of the subject based on weighted av- 
erages of the redundantly detected subj ects are ex- 
tremely effective methods for improving the detection ac- 

10 curacy, regardless of the subject to be detected. 

[Redetection Process] 

[0143] As described previously, in the case that the 
15 number of red eyes confirmed by the red eye confirming 
step 4 is one or less (none, or only one) even if the face 
detected by the face detecting step 3 is a front facing 
face, a second red eye candidate detecting step, which 
has a higher degree of accuracy than the red eye candi- 
20 date detecting step 2 is performed. Then, the face de- 
tecting step 3 and the red eye confirming step 4 are per- 
formed with respect to the red eye candidates which have 
been detected by the second red eye candidate detecting 
step. In addition, similar processes are performed in the 
25 case that the face detected in the vicinity of the red eye 
candidate is a face in profile, and number of red eye can- 
didates confirmed to be red eye by the red eye confirming 
step 4 is 0. Hereinafter, redetection of red eye candidates 
(second structural element candidate detecting process) 
30 will be described. 

[0144] There are various factors that may be consid- 
ered as reasons why red eyes are not detected as red 
eye candidates by the red eye candidate detecting step 
2. For example, the scanning density may have been 
35 reduced in order to prioritize processing speed. That is, 
detection may have been performed while moving the 
judgment target region 8 on the image 1 illustrated in 
Figure 3 in great increments (number of pixels). Alterna- 
tively, detection of red eye candidates may have been 
40 performed using a judgment target region 8 of a size (1 0 
pixels x 10 pixels, for example) appropriate for an esti- 
mated size of red eyes, yet the actual size of red eye was 
smaller than the estimated size. Thus, in the redetection 
process, the entire image is set as the search range, and 
45 a detection process, having a higher detection rate than 
the previously executed red eye candidate detecting 
process, is performed. 

[0145] As red eye detecting methods having higher de- 
tection rates than the previously executed red eye can- 
50 didate detecting step, the following may be considered. 
[01 46] First, in the case that the judgment target region 
was scanned with low scanning density in the previous 
red eye candidate detecting step, the scanning density 
of the redetection process maybe increased. At this time, 
55 scanning may be performed only on pixels that were 
scanned over in the previous red eye candidate detecting 
step. 

[0147] In addition, unlike the case in which the red eye 
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candidate detecting step is performed for the first time 
with respect to an image, facial regions are already de- 
tected by the face detecting step 3 in the redetection proc- 
ess. Accordingly, the image within the facial regions may 
be enlarged, and redetection performed with the en- 
larged facial region image as the target. 
[0148] Further, a pre-process may be administered to 
the facial region image detected by the face detecting 
step 3, in order to facilitate detection of red eye candi- 
dates, prior to redetection. 

[01 49] As a pre-process for facilitating detection of red 
eye candidates, a process for emphasizing redness in 
parts of the facial region at where redness is compara- 
tively conspicuous may be considered. For example, a 
conversion table, in which the relationships between pixel 
values are graphed as an S-shaped curve, when pixel 
values prior to the process are plotted along a horizontal 
axis, and pixel values following the process are plotted 
along a vertical axis, is prepared. Figure 25 is a diagram 
that illustrates an example of such a conversion table. 
The relationships between the pixel values prior to and 
following the process are indicated by solid line 28. Note 
that in Figure 25, broken line 29 indicates a relationship 
of "Input Values = Output Values". 
[0150] The conversion table is employed to convert 
the R, G, and B values of each of the pixels that constitute 
the facial region. In this case, if the pixel value at the point 
where the solid line 28 and the broken line 29 intersect 
is designated as T, if the R value of a pixel is greater than 
T, then the R value is converted to an even greater value. 
As a result, redness is emphasized more at regions at 
which redness is comparative conspicuous (regions 
where R>T). 

[01 51 ] As another pre-process to facilitate detection of 
red eye candidates, a process for enlarging red regions 
within facial regions may be considered. To enlarge the 
red regions, a maximum value filter maybe applied to an 
R channel, while minimum value filters are applied to a 
G channel and a B channel of the image of a facial region, 
for example. Here, the maximum value filter is a filter that 
converts the pixel values of each pixel within the filter to 
the maximum value from among the pixel values. The 
minimum value filters are filters that convert the pixel val- 
ues of each pixel within the filters to the minimum value 
from among the pixel values. By processing images with 
these types of filters, regions that include any redness 
are converted to a sharp red color. As a result, the red 
regions are enlarged. 

[01 52] The redetection process employs one or a com- 
bination of the above methods, to detect red eye candi- 
dates. The processes administered on the redetected 
red eye candidates by the the red eye confirming step 4 
are the same as those administered on red eye candi- 
dates which were detected in the first red eye candidate 
detecting step. Therefore, detailed descriptions of the 
processes will be omitted. 

[0153] It would be clear to those skilled in the art that 
the methods to improve the efficiency of red eye candi- 



date detection described previously can be applied to 
redetection process to improve the efficiency of the face 
detecting step, to a degree that does not reduce the de- 
tection accuracy thereof, compared to the detection ac- 

5 curacy of the red eye candidate detecting step. 

[0154] In this manner, the redetection process is per- 
formed only in cases that a previous red eye confirming 
step confirms one or less red eyes. Therefore, the 
processing time as a whole is reduced compared to a 

10 case in which a red eye candidate detecting method hav- 
ing a high detection rate is employed first. In addition, 
red eyes can be positively confirmed, if they exist. 

<Utilization of the Detection Results> 

15 

[01 55] The red eye detection results are utilized to cor- 
rect red eye, for example. Figure 24 illustrates an exam- 
ple of a red eye correcting process. In the exemplary 
process, first, pixels, of which the color difference value 
20 Cr exceeds a predetermined value, are extracted. Then, 
a morphology process is administered to shape the ex- 
tracted region. Finally, the colors of each pixel that con- 
stitute the shaped region are replaced with colors which 
are appropriate for pupils (such as a gray of a predeter- 
25 mined brightness). 

[0156] Note that other known methods for correcting 
red eyes within images may be applied as well. 
[01 57] An alternative embodiment may be considered 
in which red eye is not corrected, but a warning is issued 
30 indicating that a red eye phenomenon has occurred. For 
example, a red eye detecting function may be incorpo- 
rated into adigital camera. The red eye detecting process 
maybe executed on an image immediately following pho- 
tography thereof, and an alarm that suggests that pho- 
35 tography be performed again may be output from a 
speaker in the case that red eyes are detected. 
[0158] Note that in cases in which output of an alarm 
is sufficient, accurate positional coordinates of red eyes 
are not necessary. Therefore, the process of confirming 
40 the positions of red eyes may be omitted from the red 
eye confirming step 4. 

[Other Embodiments] 

45 [0159] An embodimentof the present invention, as well 
as modifications and combinations thereof, has been de- 
scribed above. However, the technical range of the 
present invention is not limited to the embodiment de- 
scribed above. Various modifications and combinations 

50 thereof are possible. 

[0160] For example, pre-processing of facial region im- 
ages, to facilitate red eye detection, has been proposed 
for the redetection process. However, this type of 
pre-process may be administered on the entire image 

55 prior to the red eye candidate detecting step 2. 

[0161] In addition, skin colored regions may be detect- 
ed from within images prior to the red eye candidate de- 
tecting step. In this case, the red eye candidate detecting 
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step can be executed with respect to only the detected 
skin colored regions. 

[0162] Further, the present invention is applicable to 
cases other than those in which eyes are photographed 
as red eyes. For example, there are cases in which hu- 
man eyes are photographed as gold, depending on the 
manner in which light is reflected thereby. In addition, 
nocturnal animals have layers that reflect light (tapetums) 
behind their retinas. In the case that light is reflected by 
tapetums, it is often the case that eyes are photographed 
as yellow-green. The present invention can also be ap- 
plied to detect eyes of these different colors. Note that 
normally colored eyes (black, in the case of Asians) can 
also be detected, in addition to abnormally colored eyes. 
In addition, the present invention is not limited to detect- 
ing eyes, and may be applied to detect noses, mouths, 
ears, and other structural elements that constitute faces. 
[01 63] Still further, the detection targets of the present 
invention are not limited to structural elements that con- 
stitute faces. For example, learning may be performed 
employing sample images of vehicles and license plates, 
and appropriate characteristic amounts and threshold 
values may be selected. In this case, the present inven- 
tion may be utilized to detect license plates from images, 
in which vehicles are pictured. It would be easily under- 
stood by those skilled in the art that the present invention 
can be applied to detect structural elements of various 
other subjects as well. 



Claims 

1 . A structural element detecting apparatus for detect- 
ing predetermined structural elements included in 
predetermined subjects from within images that in- 
clude the subjects, comprising: 

a structural element candidate detecting sec- 
tion, for detecting structural element candidates 
by discriminating characteristics inherent to the 
predetermined structural elements, from among 
characteristics of the images; 
a subject detecting section, for detecting sub- 
jects that include the structural element candi- 
dates by discriminating characteristics inherent 
to the predetermined subjects, from among 
characteristics of the images in the vicinities of 
the structural elements candidates; and 
a structural element confirming section, for con- 
firming that the structural element candidates 
are the predetermined structural elements, by 
discriminating characteristics inherent to the 
predetermined structural elements with a higher 
degree of accuracy than that of the structural 
element candidate detecting section, from 
among the characteristics of the images in the 
vicinities of the structural element candidates, 
which are included within subjects detected by 



the subject detecting section. 

2. A structural element detecting apparatus as defined 
in Claim 1 , wherein: 

5 

the subject detecting section is capable of exe- 
cuting a plurality of subject detecting processes, 
which have different detection properties; and 
a second subject detecting process, having dif- 
10 ferent detection properties from that of a first 

subject detecting process, is executed in the 
case that the first subject detecting process does 
not detect a subject. 

15 3. A structural element detecting apparatus as defined 
in either Claim 1 or Claim 2, wherein: 

the structural element confirming section per- 
forms discrimination, utilizing data regarding 
20 subjects detected by the subject detecting sec- 

tion. 

4. A structural element detecting apparatus as defined 
in Claim 3, wherein: 

25 

the structural element confirming section con- 
firms as the predetermined structural elements 
those structural element candidates which have 
characteristics inherent to structural elements 
30 of the same orientations as the orientations of 

subjects detected by the subject detecting sec- 
tion. 

5. A structural element detecting apparatus as defined 
35 in either Claim 3 or Claim 4, wherein: 

the structural element confirming section con- 
firms as the predetermined structural elements 
those structural element candidates which have 
40 characteristics inherent to structural elements 

of sizes estimated from the sizes of subjects de- 
tected by the subject detecting section. 

6. A structural element detecting apparatus as defined 
45 in any one of Claims 3 through 5, wherein: 

the structural element confirming section com- 
pares the number of structural element candi- 
dates, detected by a first structural element can- 
50 didate detecting process and confirmed as be- 

ing the predetermined structural elements, 
against the maximum number of structural ele- 
ments which are possibly included within the 
subjects detected by the subject detecting sec- 
55 tion; and 

a second structural element candidate detecting 
process, having a higher detection rate than the 
first structural element candidate detecting proc- 
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ess, is executed in the case that number of struc- 
tural element candidates, which have been de- 
tected by the first structural element candidate 
detecting process and confirmed to be structural 
elements by the structural element confirming 5 
section, is less than the maximum number. 

7. A structural element detecting apparatus as defined 
in any one of Claims 1 through 6, wherein: 

10 

the structural element confirming section con- 
firms the positions of structural elements, based 
on data obtained during the process of discrim- 
inating characteristics inherent to the predeter- 
mined structural elements, from among the 15 
characteristics of the images in the vicinity of the 
structural element candidates included in the 
subjects detected by the subject detecting sec- 
tion. 

20 

8. A structural element detecting apparatus as defined 
in any one of Claims 1 through 7, wherein the struc- 
tural element candidate detecting section compris- 
es: 

25 

target region setting means, for setting judgment 
target regions within an image; and 
judging means, forjudging whether images with- 
in the judgment target regions set by the target 
region setting means represent structural ele- 30 
ments, based on a plurality of characteristic 
amounts, which are calculated from pixel values 
of pixels that constitute the images within the 
judgment target regions; 

the judging means executing the processes of: 35 

classifying and recording formulas for cal- 
culating the plurality of characteristic 
amounts into N groups (N>1 ), such that the 
amount of calculations required to obtain 40 
characteristic amounts by formulas in a K th 
group (1<K<N) is less than the amount of 
calculations required to obtain characteris- 
tic amounts by formulas in a (K+1 ) th group; 
outputting judgment results indicating that 45 
an image within a judgment target region 
does not represent structural elements, in 
the case that itisjudged thatthe image with- 
in the judgment target region does not rep- 
resent structural elements, based on char- so 
acteristic amounts obtained by formulas in 
a K th group; 

performing judgment, based on character- 
istic amounts obtained by formulas in a 
(K+1) th group, in the case that it is judged 55 
that an image within a judgment target re- 
gion represents structural elements, based 
on characteristic amounts obtained by for- 



mulas in a K th group; and 
outputting judgment results indicating that 
the image within the judgment target region 
represents structural elements, in the case 
that it is judged that the image within the 
judgment target region represents structur- 
al elements, based on characteristic 
amounts obtained by formulas in the N th 
group. 

9. A structural element detecting apparatus as defined 
in Claim 8, wherein: 

the judging means performs first judgment, 
based on the characteristic amounts obtained 
by formulas in a K th group, with respect to a plu- 
rality of judgment target regions set by the target 
region setting means; and 
the judging means performs second judgment, 
based on the characteristic amounts obtained 
by formulas in a (K+1) th group, with respect to 
judgment target regions, which have been 
judged to represent structural elements by the 
first judgment. 

10. A structural element detecting apparatus as defined 
in either Claim 8 or Claim 9, wherein: 

the judgment target regions are set such that 
the positions of judgment target regions closest 
to each other are shifted at least two pixels, dur- 
ing at least a portion of the judgments performed 
by the judging means. 

1 1 . A structural element detecting apparatus as defined 
in either Claim 8 or Claim 9, wherein: 

the judging means performs first judgment, 
based on the characteristic amounts obtained 
by formulas in a K 111 group, with respect to the 
plurality of judgment target regions set by the 
target region setting means; and 
the judging means performs second judgment, 
based on the characteristic amounts obtained 
by formulas in a (K+1) th group, with respect to 
a plurality of judgment target regions, which are 
set such that at least a portion thereof overlaps 
with judgment target regions that have been 
judged to represent structural elements by the 
first judgment. 

12. A structural element detecting apparatus as defined 
in either Claim 1 0 or Claim 1 1 , wherein: 

the judging means performs first judgment, 
based on characteristic amounts obtained by 
formulas in a K th group, with respect to judgment 
target regions, which are set within an image 
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represented at a predetermined resolution; and 
the judging means performs second judgment, 
based on characteristic amounts obtained by 
formulas in a (K+1) th group, with respect to the 
judgment target regions, which are set within the 5 
image represented at a higher resolution than 
that of the first judgment. 

13. A structural element detecting apparatus as defined 

in Claim 12, wherein: 10 

at least one of the formulas for calculating char- 
acteristic amounts classified in a K th group is 
also included in a (K+1) th group. 



predetermined subject, based on charac- 
teristic amounts obtained by formulas in the 
N th group. 

15. A structural element detecting apparatus as defined 
in Claim 14, wherein: 

the judging means performs first judgment, 
based on the characteristic amounts obtained 
by formulas in a K th group, with respect to a plu- 
rality of judgment target regions set by the target 
region setting means; and 
the judging means performs second judgment, 
based on the characteristic amounts obtained 
by formulas in a (K+1) th group, with respect to 
judgment target regions, which have been 
judged to represent structural elements by the 
first judgment. 



15 

A structural element detecting apparatus as defined 
in any one of Claims 1 through 13, wherein the sub- 
ject detecting section comprises: 



target region setting means, forsetting judgment 20 
target regions within an image; and 
judging means, forjudging whether images with- 
in the judgment target regions set by the target 
region setting means represent the predeter- 
mined subjects, based on a plurality of charac- 25 
teristic amounts, which are calculated from pixel 
values of pixels that constitute the images within 
the judgment target regions; 
the judging means executing the processes of: 

30 

classifying and recording formulas for cal- 
culating the plurality of characteristic 
amounts into N groups (N>1 ), such that the 
amount of calculations required to obtain 
characteristic amounts by formulas in a K th 35 
group (1<K<N) is less than the amount of 
calculations required to obtain characteris- 
tic amounts by formulas in a (K+1) th group; 
outputting judgment results indicating that 
an image within a judgment target region 40 
does not represent the predetermined sub- 
ject, in the case that it is judged that the 
image within the judgment target region 
does not represent the predetermined sub- 
ject, based on characteristic amounts ob- 45 
tained by formulas in a K th group; 
performing judgment, based on character- 
istic amounts obtained by formulas in a 
(K+1) th group, in the case that it is judged 
that an image within a judgment target re- so 
gion represents the predetermined subject, 
based on characteristic amounts obtained 
by formulas in a K th group; and 
outputting judgment results indicating that 
the image within the judgment target region 55 
represents the predetermined subject, in 
the case that itisjudged thatthe image with- 
in the judgment target region represents the 



16. A structural element detecting apparatus as defined 
in either Claim 14 or Claim 15, wherein: 

the judgment target regions are set such that 
the positions of judgment target regions closest 
to each other are shifted at least two pixels, dur- 
ing at least a portion of the judgments performed 
by the judging means. 

17. A structural element detecting apparatus as defined 
in either Claim 14 or Claim 15, wherein: 

the judging means performs first judgment, 
based on characteristic amounts obtained by 
formulas in a K th group, with respect to judgment 
target regions, which are set within an image 
represented at a predetermined resolution; and 
the judging means performs second judgment, 
based on characteristic amounts obtained by 
formulas in a (K+1 ) th group, with respect to the 
judgmenttarget regions, which aresetwithin the 
image represented at a higher resolution than 
that of the first judgment. 

18. A structural element detecting apparatus as defined 
in either Claim 1 6 or Claim 1 7, wherein: 

the judging means performs first judgment, 
based on the characteristic amounts obtained 
by formulas in a K xh group, with respect to the 
plurality of judgment target regions set by the 
target region setting means; and 
the judging means performs second judgment, 
based on the characteristic amounts obtained 
by formulas in a (K+1) th group, with respect to 
a plurality of judgment target regions, which are 
set such that at least a portion thereof overlaps 
with judgment target regions that have been 
judged to represent structural elements by the 
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first judgment. 

19. A structural element detecting apparatus as defined 
in Claim 18, wherein: 

5 

at least one of the formulas for calculating char- 
acteristic amounts classified in a K th group is 
also included in a (K+1) th group. 

20. A structural element detecting apparatus as defined 10 
in any one of Claims 1 through 19, wherein: 

the structural elements are eyes of living organ- 
isms. 

15 

21 . A structural element detecting apparatus as defined 
in Claim 20, wherein: 

the colors of the eyes are abnormal colors. 

20 

22. A structural element detecting apparatus as defined 
in either Claim 20 or Claim 21 , wherein: 

the predetermined subjects are faces of the liv- 
ing organisms. 25 

23. A detecting apparatus, comprising: 

target region setting means, for setting judgment 
target regions within an image; and 30 
judging means, forjudging whether images with- 
in the judgment target regions set by the target 
region setting means represent detection tar- 
gets, based on a plurality of characteristic 
amounts, which are calculated from pixel values 35 
of pixels that constitute the images within the 
judgment target regions; 

the judging means executing the processes of: 

classifying and recording formulas for cal- 40 
culating the plurality of characteristic 
amounts into N groups (N>1 ), such that the 
amount of calculations required to obtain 
characteristic amounts by formulas in a K th 
group (1<K<N) is less than the amount of 45 
calculations required to obtain characteris- 
tic amounts by formulas in a (K+1 )th group; 
outputting judgment results indicating that 
an image within a judgment target region 
does not represent the detection targets, in 50 
the case that itisjudged thatthe image with- 
in the judgment target region does not rep- 
resent the detection targets, based on char- 
acteristic amounts obtained by formulas in 
a K th group; 55 
performing judgment, based on character- 
istic amounts obtained by formulas in a 
(K+1)th group, in the case that it is judged 



that an image within a judgment target re- 
gion represents the detection targets, 
based on characteristic amounts obtained 
by formulas in a K th group; and 
outputting judgment results indicating that 
the image within the judgment target region 
represents the detection targets, in the case 
that it is judged that the image within the 
judgment target region represents the de- 
tection targets, based on characteristic 
amounts obtained by formulas in the N th 
group. 

24. A structural element detecting method for detecting 
predetermined structural elements included in pre- 
determined subjects from within images that include 
the subjects, comprising: 

a first structural element candidate detecting 
process, for detecting structural element candi- 
dates by discriminating characteristics inherent 
to the predetermined structural elements, from 
among characteristics of the images; 
a first subject detecting process, for detecting 
subjects that include the structural element can- 
didates by discriminating characteristics inher- 
ent to the predetermined subjects, from among 
characteristics of the images in the vicinities of 
the structural element candidates; and 
astructural elementconfirming process, forcon- 
firming that the structural element candidates 
are the predetermined structural elements, by 
discriminating characteristics inherent to the 
predetermined structural elements with a higher 
degree of accuracy than that of the structural 
element candidate detecting section, from 
among the characteristics of the images in the 
vicinities of the structural element candidates, 
which are included within the subjects detected 
by the subject detecting section. 

25. A structural element detecting method as defined in 
Claim 24, wherein: 

a second subject detecting process, having dif- 
ferent detection properties from that of a first 
subject detecting process, is executed in the 
case that the first subject detecting process does 
not detect a subject. 

26. A structural element detecting method as defined in 
either Claim 24 or Claim 25, wherein: 

the structural element confirming process per- 
forms discrimination, utilizing data regarding 
subjects detected by the subject detecting proc- 
ess. 
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27. A structural element detecting method as defined in 
Claim 26, wherein: 

the structural element confirming process con- 
firms as the predetermined structural elements 
those structural element candidates which have 
characteristics inherent to structural elements 
of the same orientations as the orientations of 
subjects detected in the subject detecting proc- 
ess. 

28. A structural element detecting method as defined in 
either Claim 26 or Claim 27, wherein: 

the structural element confirming process con- 
firms as the predetermined structural elements 
those structural element candidates which have 
characteristics inherent to structural elements 
of sizes estimated from the sizes of subjects de- 
tected in the subject detecting process. 

29. A structural element detecting method as defined in 
any one of Claims 26 through 28, wherein: 

the number of structural element candidates, 
detected by a first structural element candidate 
detecting process and confirmed as being the 
predetermined structural elements, is compared 
against the maximum number of structural ele- 
ments which are possibly included within the 
subjects detected by the subject detecting sec- 
tion; 

a second structural element candidate detecting 
process, having a higher detection rate than the 
first structural element candidate detecting proc- 
ess, is executed in the case that number of struc- 
tural element candidates, which have been de- 
tected by the first structural element candidate 
detecting process and confirmed to be structural 
elements by the structural element confirming 
section, is less than the maximum number; and 
the structural element confirming process is ex- 
ecuted with respect to the structural element 
candidates which have been detected by the 
second structural element candidate detecting 
process. 

30. A structural element detecting method as defined in 
any one of Claims 24 through 29, wherein: 

the structural element confirming process con- 
firms the positions of structural elements, based 
on data obtained during the process of discrim- 
inating characteristics inherent to the predeter- 
mined structural elements, from among the 
characteristics of the images in the vicinity of the 
structural element candidates included in the 
subjects detected by the subject detecting proc- 



ess. 

31 . A structural element detecting method as defined in 
any one of Claims 24 through 30, wherein the first 

5 and/or second structural element candidate detect- 

ing process comprises: 

a target region setting step, for setting judgment 
target regions within an image; and 

10 ajudging step, forjudging whether images within 

the judgment target regions set in the target re- 
gion setting step represent structural elements, 
based on a plurality of characteristic amounts, 
which are calculated from pixel values of pixels 

15 that constitute the images within the judgment 

target regions; 

the judging step comprising the processes of: 

classifying and recording formulas for cal- 
20 culating the plurality of characteristic 

amounts into N groups (N>1 ), such that the 
amount of calculations required to obtain 
characteristic amounts by formulas in a K th 
group (1<K<N) is less than the amount of 
25 calculations required to obtain characteris- 

tic amounts by formulas in a (K+1 ) th group; 
outputting judgment results indicating that 
an image within a judgment target region 
does not represent structural elements, in 
30 the case that it is judged that the image with- 

in the judgment target region does not rep- 
resent structural elements, based on char- 
acteristic amounts obtained by formulas in 
a K th group; 

35 performing judgment, based on character- 

istic amounts obtained by formulas in a 
(K+1) th group, in the case that it is judged 
that an image within a judgment target re- 
gion represents structural elements, based 
40 on characteristic amounts obtained by for- 

mulas in a K th group; and 
outputting judgment results indicating that 
the image within the judgment target region 
represents structural elements, in the case 
45 that it is judged that the image within the 

judgment target region represents structur- 
al elements, based on characteristic 
amounts obtained by formulas in the N th 
group. 

50 

32. A structural element detecting method as defined in 
Claim 31 , wherein: 

first judgment, based on the characteristic 
55 amounts obtained by formulas in a K th group, is 

performed with respect to a plurality of judgment 
target regions set in the target region setting 
step; and 
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second judgment, based on the characteristic 
amounts obtained by formulas in a (K+1) 1h 
group, is performed with respect to judgment tar- 
get regions, which have been judged to repre- 
sent structural elements by the first judgment. 5 

33. A structural element detecting method as defined in 
either Claim 31 or Claim 32, wherein: 

the judgment target regions are set such that 10 
the positions of judgment target regions closest 
to each other are shifted at least two pixels, dur- 
ing at least a portion of the judgments performed 
in the judging step. 

15 

34. A structural element detecting method as defined in 
either Claim 31 or Claim 32, wherein: 

first judgment, based on characteristic amounts 
obtained by formulas in a K th group, is performed 20 
with respect to judgment target regions, which 
are set within an image represented at a prede- 
termined resolution; and 

second judgment, based on characteristic 
amounts obtained by formulas in a (K+1) th 25 
group, is performed with respectto the judgment 
target regions, which are set within the image 
represented at a higher resolution than that of 
the first judgment. 

30 

35. A structural element detecting method as defined in 
either Claim 33 or Claim 34, wherein: 



36. A structural element detecting method as defined in 
Claim 39, wherein: 



37. A structural element detecting method as defined in 

any one of Claims 24 through 36, wherein the first 55 
and/or second subject detecting process comprises: 

a target region setting step, for setting judgment 



target regions within an image; and 
ajudging step, forjudging whether images within 
the judgment target regions set in the target re- 
gion setting step represent the predetermined 
subjects, based on a plurality of characteristic 
amounts, which are calculated from pixel values 
of pixels that constitute the images within the 
judgment target regions; 
the judging step comprising the processes of: 

classifying and recording formulas for cal- 
culating the plurality of characteristic 
amounts into N groups (N>1 ), such that the 
amount of calculations required to obtain 
characteristic amounts by formulas in a K th 
group (1<K<N) is less than the amount of 
calculations required to obtain characteris- 
tic amounts by formulas in a (K+1 ) th group; 
outputting judgment results indicating that 
an image within a judgment target region 
does not represent the predetermined sub- 
ject, in the case that it is judged that the 
image within the judgment target region 
does not represent the predetermined sub- 
ject, based on characteristic amounts ob- 
tained by formulas in a K th group; 
performing judgment, based on character- 
istic amounts obtained by formulas in a 
(K+1) th group, in the case that it is judged 
that an image within a judgment target re- 
gion represents the predetermined subject, 
based on characteristic amounts obtained 
by formulas in a K th group; and 
outputting judgment results indicating that 
the image within the judgment target region 
represents the predetermined subject, in 
the case that it is judged that the image with- 
in the judgment target region represents the 
predetermined subject, based on charac- 
teristic amounts obtained by formulas in the 
N th group. 

38. A structural element detecting method as defined in 
Claim 37, wherein: 

first judgment, based on the characteristic 
amounts obtained by formulas in a K th group, is 
performed with respectto a plurality of judgment 
target regions set in the target region setting 
step; and 

second judgment, based on the characteristic 
amounts obtained by formulas in a (K+1) th 
group, is performed with respect to judgment tar- 
get regions, which have been judged to repre- 
sent structural elements by the first judgment, 
in the subject detecting process. 

39. A structural element detecting method as defined in 



10 



15 



first judgment, based on the characteristic 
amounts obtained by formulas in a K th group, is 35 
performed with respect to the plurality of judg- 
ment target regions set in the target region set- 
ting step; and 

second judgment, based on the characteristic 
amounts obtained by formulas in a (K+1) th 
group, is performed with respect to a plurality of 
judgment target regions, which are set such that 
at least a portion thereof overlaps with judgment 
target regions that have been judged to repre- 
sent structural elements by the first judgment. 45 



at least one of the formulas for calculating char- 50 
acteristic amounts classified in a K th group is 
also included in a (K+1) th group. 
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either Claim 37 or Claim 38, wherein: 

the judgment target regions are set such that 
the positions of judgment target regions closest 
to each other are shifted at least two pixels, dur- 5 
ing at least a portion of the judgments performed 
in the judging step, in the subject detecting proc- 
ess. 

40. A structural element detecting method as defined in 10 
either Claim 37 or Claim 38, wherein: 

first judgment, based on characteristic amounts 
obtained by formulas in aK th group, is performed 
with respect to judgment target regions, which 15 
are set within an image represented at a prede- 
termined resolution; and 

second judgment, based on characteristic 
amounts obtained by formulas in a (K+1) th 
group, is performed with respectto the judgment 20 
target regions, which are set within the image 
represented at a higher resolution than that of 
the first judgment, in the subject detecting proc- 
ess. 

25 

41 . A structural element detecting method as defined in 
either Claim 39 or Claim 40, wherein: 

first judgment, based on the characteristic 
amounts obtained by formulas in a K th group, is 30 
performed with respect to the plurality of judg- 
ment target regions set in the target region set- 
ting step; and 

second judgment, based on the characteristic 
amounts obtained by formulas in a (K+1) th 35 
group, is performed with respect to a plurality of 
judgment target regions, which are set such that 
at least a portion thereof overlaps with judgment 
target regions that have been judged to repre- 
sent structural elements by the first judgment, 40 
in the subject detecting process. 

42. A structural element detecting method as defined in 
Claim 41 , wherein: 

45 

at least one of the formulas for calculating char- 
acteristic amounts classified in a K th group is 
also included in a (K+1) th group, in the subject 
detecting process. 

50 

43. A structural element detecting method as defined in 
any one of Claims 24 through 42, wherein: 

the structural elements are eyes of living organ- 
isms. 55 

44. A structural element detecting method as defined in 
Claim 43, wherein: 



the colors of the eyes are abnormal colors. 

45. A structural element detecting method as defined in 
either Claim 43 or Claim 44, wherein: 

the predetermined subjects are faces of the liv- 
ing organisms. 

46. A detecting method, comprising: 

a target region setting step, for setting judgment 
target regions within an image; and 
ajudgingstep, forjudging whether images within 
the judgment target regions set in the target re- 
gion setting step represent detection targets, 
based on a plurality of characteristic amounts, 
which are calculated from pixel values of pixels 
that constitute the images within the judgment 
target regions; 

the judging step executing the processes of: 

classifying and recording formulas for cal- 
culating the plurality of characteristic 
amounts into N groups (N>1 ), such that the 
amount of calculations required to obtain 
characteristic amounts by formulas in a K th 
group (1<K<N) is less than the amount of 
calculations required to obtain characteris- 
tic amounts by formulas in a (K+1 )th group; 
outputting judgment results indicating that 
an image within a judgment target region 
does not represent the detection targets, in 
thecasethat it is judged that the image with- 
in the judgment target region does not rep- 
resent the detection targets, based on char- 
acteristic amounts obtained by formulas in 
a K th group; 

performing judgment, based on character- 
istic amounts obtained by formulas in a 
(K+1 ) th group, in the case that it is judged 
that an image within a judgment target re- 
gion represents the detection targets, 
based on characteristic amounts obtained 
by formulas in a K th group; and 
outputting judgment results indicating that 
the image within the judgment target region 
represents the detection targets, in the case 
that it is judged that the image within the 
judgment target region represents the de- 
tection targets, based on characteristic 
amounts obtained by formulas in the N th 
group. 

47. A computer readable recording medium having a 
program that provides a function of detecting prede- 
termined structural elements included in predeter- 
mined subjects from within images that include the 
subjects, the program causing a computer to exe- 
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cute: 

a first structural element candidate detecting 
process, for detecting structural element candi- 
dates by discriminating characteristics inherent 5 
to the predetermined structural elements, from 
among characteristics of the images; 
a first subject detecting process, for detecting 
subjects that include the structural element can- 
didates by discriminating characteristics inher- 10 
ent to the predetermined subjects, from among 
characteristics of the images in the vicinities of 
the structural element candidates; and 
a structural element confirming process, for con- 
firming that the structural element candidates is 
are the predetermined structural elements, by 
discriminating characteristics inherent to the 
predetermined structural elements with a higher 
degree of accuracy than that of the structural 
element candidate detecting section, from 20 
among the characteristics of the images in the 
vicinities of the structural element candidates, 
which are included within the subjects detected 
by the subject detecting section. 

25 

48. A recording medium as defined in Claim 47, wherein 
the program recorded therein causes a computer to 
execute: 

a second subject detecting process, having dif- 30 
ferent detection properties from that of a first 
subject detecting process, in the case that the 
first subject detecting process does not detect 
a subject. 

35 

49. A recording medium as defined in either Claim 47 or 
Claim 48, wherein: 

the structural element confirming process per- 
forms discrimination, utilizing data regarding 40 
subjects detected by the subject detecting proc- 
ess. 

50. A recording medium as defined in Claim 49, wherein: 

45 

the structural element confirming process con- 
firms as the predetermined structural elements 
those structural element candidates which have 
characteristics inherent to structural elements 
of the same orientations as the orientations of 50 
subjects detected in the subject detecting proc- 
ess. 

51. A recording medium as defined in either Claim 49 or 
Claim 50, wherein: 55 

the structural element confirming process con- 
firms as the predetermined structural elements 



those structural element candidates which have 
characteristics inherent to structural elements 
of sizes estimated from the sizes of subjects de- 
tected in the subject detecting process. 

52. A recording medium as defined in any one of Claims 
49 through 51, wherein: 

the number of structural element candidates, 
detected by a first structural element candidate 
detecting process and confirmed as being the 
predetermined structural elements, is compared 
against the maximum number of structural ele- 
ments which are possibly included within the 
subjects detected by the subject detecting sec- 
tion; 

a second structural element candidate detecting 
process, having a higher detection rate than the 
first structural element candidate detecting proc- 
ess, is executed in the case that number of struc- 
tural element candidates, which have been de- 
tected by the first structural element candidate 
detecting process and confirmed to be structural 
elements by the structural element confirming 
section, is less than the maximum number; and 
the structural element confirming process is ex- 
ecuted with respect to the structural element 
candidates which have been detected by the 
second structural element candidate detecting 
process. 

53. A recording medium as defined in any one of Claims 
47 through 52, wherein: the structural element con- 
firming process confirms structural element candi- 
dates as structural elements, based on data obtained 
during the process of discriminating characteristics 
inherent to pupils that have regions displayed red, 
from among the characteristics of the images in the 
vicinity of the structural element candidates included 
in the subjects detected by the subject detecting 
process. 

54. A recording medium as defined in any one of Claims 
47 through 53, wherein the first and/or second struc- 
tural element candidate detecting process compris- 
es: 

a target region setting step, for setting judgment 
target regions within an image; and 
ajudging step, forjudging whether images within 
the judgment target regions set in the target re- 
gion setting step represent structural elements, 
based on a plurality of characteristic amounts, 
which are calculated from pixel values of pixels 
that constitute the images within the judgment 
target regions; 

the judging step comprising the processes of: 
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classifying and recording formulas for cal- 
culating the plurality of characteristic 
amounts into N groups (N>1 ), such that the 
amount of calculations required to obtain 
characteristic amounts by formulas in a K th 
group (1<K<N) is less than the amount of 
calculations required to obtain characteris- 
tic amounts by formulas in a (K+1) th group; 
outputting judgment results indicating that 
an image within a judgment target region 
does not represent structural elements, in 
the case that it is judged thatthe image with- 
in the judgment target region does not rep- 
resent structural elements, based on char- 
acteristic amounts obtained by formulas in 
a K th group; 

performing judgment, based on character- 
istic amounts obtained by formulas in a 
(K+1) th group, in the case that it is judged 
that an image within a judgment target re- 
gion represents structural elements, based 
on characteristic amounts obtained by for- 
mulas in a K th group; and 
outputting judgment results indicating that 
the image within the judgment target region 
represents structural elements, in the case 
that it is judged that the image within the 
judgment target region represents structur- 
al elements, based on characteristic 
amounts obtained by formulas in the N th 
group. 

55. A recording medium as defined in Claim 54, wherein: 

first judgment, based on the characteristic 
amounts obtained by formulas in a K th group, is 
performed with respect to a plurality of judgment 
target regions set in the target region setting 
step; and 

second judgment, based on the characteristic 
amounts obtained by formulas in a (K+1) th 
group, is performed with respecttojudgmenttar- 
get regions, which have been judged to repre- 
sent structural elements by the first judgment. 

56. A recording medium as defined in either Claim 54 or 
Claim 55, wherein: 



obtained by formulas in a K th group, is performed 
with respect to judgment target regions, which 
are set within an image represented at a prede- 
termined resolution; and 

5 second judgment, based on characteristic 

amounts obtained by formulas in a (K+1) th 
group, is performed with respect to the judgment 
target regions, which are set within the image 
represented at a higher resolution than that of 

10 the first judgment. 

58. A recording medium as defined in either Claim 56 or 
Claim 57, wherein: 

15 first judgment, based on the characteristic 

amounts obtained by formulas in a K th group, is 
performed with respect to the plurality of judg- 
ment target regions set in the target region set- 
ting step; and 

20 second judgment, based on the characteristic 

amounts obtained by formulas in a (K+1) th 
group, is performed with respect to a plurality of 
judgment target regions, which are set such that 
at least a portion thereof overlaps with judgment 

25 target regions that have been judged to repre- 

sent structural elements by the first judgment. 

59. A recording medium as defined in Claim 58, wherein: 

30 at least one of the formulas for calculating char- 

acteristic amounts classified in a K th group is 
also included in a (K+1) th group. 

60. A recording medium as defined in any one of Claims 
35 47 through 59, wherein the first and/or second sub- 
ject detecting process comprises: 

a target region setting step, for setting judgment 
target regions within an image; and 

40 ajudging step, forjudging whether images within 

the judgment target regions set in the target re- 
gion setting step represent the predetermined 
subjects, based on a plurality of characteristic 
amounts, which are calculated from pixel values 

45 of pixels that constitute the images within the 

judgment target regions; 
the judging step comprising the processes of: 



the judgment target regions are set such that 
the positions of judgment target regions closest 50 
to each other are shifted at least two pixels, dur- 
ing at least a portion of the judgments performed 
in the judging step. 

57. A recording medium as defined in either Claim 54 or 55 
Claim 55, wherein: 

first judgment, based on characteristic amounts 



classifying and recording formulas for cal- 
culating the plurality of characteristic 
amounts into N groups (N>1 ), such that the 
amount of calculations required to obtain 
characteristic amounts by formulas in a K th 
group (1<K<N) is less than the amount of 
calculations required to obtain characteris- 
tic amounts by formulas in a (K+1 ) th group; 
outputting judgment results indicating that 
an image within a judgment target region 
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does not represent the predetermined sub- 
ject, in the case that it is judged that the 
image within the judgment target region 
does not represent the predetermined sub- 
ject, based on characteristic amounts ob- 
tained by formulas in a K th group; 
performing judgment, based on character- 
istic amounts obtained by formulas in a 
(K+1) th group, in the case that it is judged 
that an image within a judgment target re- 
gion represents the predetermined subject, 
based on characteristic amounts obtained 
by formulas in a K th group; and 
outputting judgment results indicating that 
the image within the judgment target region 
represents the predetermined subject, in 
the case that itisjudged thatthe image with- 
in the judgment target region represents the 
predetermined subject, based on charac- 
teristic amounts obtained by formulas in the 
N th group. 

61 . A recording medium as defined in Claim 60, wherein: 

first judgment, based on the characteristic 
amounts obtained by formulas in a K th group, is 
performed with respect to a plurality of judgment 
target regions set in the target region setting 
step; and 

second judgment, based on the characteristic 
amounts obtained by formulas in a (K+1) th 
group, is performed with respecttojudgmenttar- 
get regions, which have been judged to repre- 
sent structural elements by the first judgment, 
in the subject detecting process. 



represented at a higher resolution than that of 
the first judgment, in the subject detecting proc- 
ess. 

5 64. A recording medium as defined in either Claim 62 or 
Claim 63, wherein: 

first judgment, based on the characteristic 
amounts obtained by formulas in a K th group, is 
10 performed with respect to the plurality of judg- 

ment target regions set in the target region set- 
ting step; and 

second judgment, based on the characteristic 
amounts obtained by formulas in a (K+1) th 

15 group, is performed with respect to a plurality of 

judgment target regions, which are set such that 
at least a portion thereof overlaps with judgment 
target regions that have been judged to repre- 
sent structural elements by the first judgment, 

20 in the subject detecting process. 

65. A recording medium as defined in Claim 77, wherein: 

at least one of the formulas for calculating char- 
ts acteristic amounts classified in a K th group is 
also included in a (K+1) th group, in the subject 
detecting process. 

66. A recording medium as defined in any one of Claims 
30 47 through 65, wherein: 

the structural elements are eyes of living organ- 
isms. 

35 67. A recording medium as defined in Claim 66, wherein: 



62. A recording medium as defined in either Claim 60 or 
Claim 61 , wherein: 

the judgment target regions are set such that 
the positions of judgment target regions closest 
to each other are shifted at least two pixels, dur- 
ing at least a portion of the judgments performed 
in the judging step, in the subject detecting proc- 
ess. 

63. A recording medium as defined in either Claim 60 or 
Claim 61 , wherein: 



the colors of the eyes are abnormal colors. 

68. A recording medium as defined in either Claim 66 or 
^0 Claim 67, wherein: 

the predetermined subjects are faces of the liv- 
ing organisms. 

45 69. A computer readable recording medium having a 
program that provides a function of detecting detec- 
tion targets included in judgment target regions, the 
program causing a computer to execute: 



first judgment, based on characteristic amounts 
obtained by formulas in a K th group, is performed 
with respect to judgment target regions, which 
are set within an image represented at a prede- 
termined resolution; and 

second judgment, based on characteristic 
amounts obtained by formulas in a (K+1) th 
group, is performed with respectto the judgment 
target regions, which are set within the image 



50 a target region setting step, for setting the judg- 

ment target regions within an image; and 
ajudging step, forjudging whether images within 
the judgment target regions set in the target re- 
gion setting step represent detection targets, 

55 based on a plurality of characteristic amounts, 

which are calculated from pixel values of pixels 
that constitute the images within the judgment 
target regions; 
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the judging step executing the processes of: 

classifying and recording formulas for cal- 
culating the plurality of characteristic 
amounts into N groups (N>1 ), such that the 5 
amount of calculations required to obtain 
characteristic amounts by formulas in a K th 
group (1<K<N) is less than the amount of 
calculations required to obtain characteris- 
tic amounts by formulas in a (K+1 )th group; 10 
outputting judgment results indicating that 
an image within a judgment target region 
does not represent the detection targets, in 
the case that it is judged thatthe image with- 
in the judgment target region does not rep- 15 
resent the detection targets, based on char- 
acteristic amounts obtained by formulas in 
a K th group; 

performing judgment, based on character- 
istic amounts obtained by formulas in a 20 
(K+1)th group, in the case that it is judged 
that an image within a judgment target re- 
gion represents the detection targets, 
based on characteristic amounts obtained 
by formulas in a K th group; and 25 
outputting judgment results indicating that 
the image within the judgment target region 
represents the detection targets, in the case 
that it is judged that the image within the 
judgment target region represents the de- 30 
tection targets, based on characteristic 
amounts obtained by formulas in the N th 
group. 
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